Embedded instructions for voice user interface

ABSTRACT

Methods and systems for rendering lists of instructions and performing actions associated with those lists are described herein. In some embodiments, an individual may request that a voice activated electronic device associated with their user account assist in performing a task using a list of instructions. The list of instructions may include metadata that indicates actions capable of being performed by additional Internet of Things (“IoT”) devices. When the instructions are rendered, an instructions speechlet may recognize the metadata and may cause one or more of the IoT devices to perform a particular action. Furthermore, the metadata may also correspond to content capable of being rendered by the voice activated electronic device to assist the individual in performing a particular step of the instructions.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims priority to, U.S.patent application Ser. No. 16/158,428, entitled “EMBEDDED INSTRUCTIONSFOR VOICE USER INTERFACE,” filed on Oct. 12, 2018, which is acontinuation of, and claims priority to, U.S. patent application Ser.No. 15/473,818, entitled “EMBEDDED INSTRUCTIONS FOR VOICE USERINTERFACE,” filed on Mar. 30, 2017, in the name of Minoj Sindhwani, andissued as U.S. Pat. No. 10,102,855, on Oct. 16, 2018. The aboveapplications are hereby incorporated by reference in their entirety.

BACKGROUND

Electronic devices, such as voice activated electronic devices, arecapable of performing various functions. Some voice activated electronicdevice may also include displays capable of outputting content and/orcommunicating with one or more additional devices/systems.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustrative diagram of an exemplary system for causing anaction to be performed by a first device based recipe data provided to asecond device, in accordance with various embodiments;

FIGS. 2A and 2B are illustrative diagrams corresponding to the exemplarysystem architecture of FIG. 1 , in accordance with various embodiments;

FIG. 3A is an illustrative diagram of an exemplary system for causingsupplemental content to be rendered by a device based on content that isbeing rendered by the device, in accordance with various embodiments;

FIG. 3B is an illustrative diagram of an exemplary system fordetermining that an item associated with a list of instructions isabsent, and causing that item to be ordered, in accordance with variousembodiments;

FIGS. 4A and 4B are illustrative flowcharts of an exemplary process fordetermining an action to be performed by a first device based on listdata and audio data output by a second device, in accordance withvarious embodiments;

FIG. 5 is an illustrative diagram of an exemplary graphical userinterface including an instructions builder for generating instructiondata, in accordance with various embodiments; and

FIG. 6 is an illustrative flowchart of an exemplary process fordetermining whether an utterance corresponds to an expected utteranceassociated with instruction data representing an instruction of a listof instructions, in accordance with various embodiments.

DETAILED DESCRIPTION

The present disclosure, as set forth below, is generally directed tovarious embodiments of methods and systems generating interactiveinstructions, and using such interactive instructions to assist incompleting a task. For example, recipe data may be generated by a recipebuilder, and individuals may render the recipe data using theirelectronic devices to prepare a food item. In some embodiments, a devicemay render content including a list of instructions. List datarepresenting the list may include instructions data representing aninstruction, which may also be referred to herein as a step of the list,to be carried out in order to complete a task (e.g., preparing a fooditem). For instance, a recipe may include multiple steps to be performedto cook a food item. Additionally, the instructions data may includetext data that is intended for presentation to a human, via audiblyand/or visually, and that describes an action for the human to do tothat is associated with the task. Furthermore, in some embodiment, theinstruction data may include metadata. The metadata may be associatedwith a particular instruction, and may indicate an action to beperformed for that step, types of devices needed for the action, and/orsupplemental information associated with performing that action. Asanother example, the instructions data may include metadata thatindicates one or more items that may be needed for completing the task.

As described herein, a list of instruction may corresponding to one ormore instructions to be performed/carried out in order to complete atask. A recipe may correspond to one type of list of instruction, forinstance, that is associated with preparing a food item. For example, apizza recipe may correspond to a list of instructions whose task relatesto cooking a pizza. The recipe may include steps, which may correspondto instructions, to be performed/carried out in order to complete thetask associated with the recipe (e.g., preparing and/or cooking the fooditem).

In a non-limiting embodiment, an individual may speak an utteranceindicating that he/she would like to perform a task. For example, anindividual may say, “I want to make pizza,” or “I want to build a birdhouse.” An electronic device located within a same environment as theindividual may be capable detecting the utterance, and may send audiodata representing the utterance to a computing system to assist inperforming some or all of the task. The computing system, which mayinclude a speech-processing system, may generate text data representingthe audio data, and may determine an intent of the utterance usingnatural language understanding processing.

As an illustrative example, the intent of the utterance, “I want to makepizza,” may be for recipe data representing a pizza to be provided tothe electronic device such that the individual may follow the recipe tomake the item (e.g., a pizza). After identifying an appropriate recipe,the computing system may provide the electronic device with display datarepresenting content (e.g., a graphical user interface (“GUI”))including some or all of the instructions. For example, the recipe datamay include first step data representing a first step indicating that anoven should be pre-heated to a first temperature. In some embodiments,the first step data may include text data that is intended forpresentation to a human (e.g., the individual) and that describes afirst action (e.g., to pre-heat an oven to a first temperature) for thehuman to do for the recipe. The computing system may determine whetherthere are any additional devices associated with a same user account asthe electronic device capable of performing the desired function (e.g.,heat an oven). If so, then command data representing an instruction tocause the oven to perform the action (e.g., heat to the firsttemperature) may be generated and sent to the identified additionaldevice (e.g., an oven). Further still, the computing system may generateand send audio data representing a message indicating to the individualthat the first step is being performed. However, if the computing systemdetermines that there are no additional electronic devices associatedwith the user account that are capable of performing the first action,then the computing system may determine text data representing a messageindicating that the action that the human is to person (e.g., theindividual pre-heating their oven to the desired temperature), and maygenerate and send audio data representing the text data to theelectronic device.

In some embodiments, instruction metadata associated with instructiondata of the list of instructions may be used to identify supplementalcontent to provide to an electronic device for that particular step. Forinstance, while trying to perform a particular instruction, anindividual may inquire as to how that instruction is performed. Forexample, if one step of the pizza recipe is to “knead the dough,” theindividual may ask, “How do I knead dough?” Metadata associated withthat step's step data may be pre-populated to include supplementalinformation, such as, but not limited to, one or more contextual links,video links, and/or audio links, that are capable of being provided tothe individual to help/assist (e.g., visually and/or audibly) withperforming/carrying out that step. Continuing the example above, inresponse to receiving audio data representing the utterance “How do Iknead dough?”, the computing system may generate text data representingthe utterance. The recipe data may include metadata associated with the“knead dough” step that may indicate expected utterance text datarepresenting one or more utterances (e.g., “expected utterances”) thatmay be expected to be spoken for this step. The computing system maygenerate a similarity score between the text data and expected utterancetext data representing an expected utterance. If the similarity score isgreater than predefined threshold value, then this may indicate that theutterance is one of the expected utterances associated with theparticular step. In this particular scenario, metadata associated withthat step, such as information to assist the individual in performingone or more actions associated with the step, may be provided to theelectronic device. As an illustrative example, a URL to a videoexplaining and showing “how to knead dough” may be sent to theelectronic device. If, however, the similarity score is determined to beless than or equal to the predefined threshold value, then the computingsystem may provide the text data to a natural language understandingsystem to allow additional speech processing to be performed.

In some embodiments, notification data indicating that the electronicdevice is currently rendering content (e.g., a GUI including the recipe)thereon may be received by the computing system. Entity datarepresenting the content may be received from a domain responsible forproviding the content, and natural language understanding processing mayuse the entity data and the text data representing an utterance todetermine an intent of the utterance. For instance, because the entitydata indicates that the step of kneading dough is currently beingdisplayed, the computing system may recognize that there is metadataassociated with that step. Therefore, in response to the inquiry, thecomputing system may use the metadata to provide a video, for example,of how to knead dough to the electronic device.

In still some additional embodiments, instructions data may includemetadata indicating items needed to complete the task (e.g., ingredientsneeded to cook a food item). The computing system may determine a useraccount that the electronic device that requested the instructions datais associated with, and may determine that an additional device capableof storing, tracking, and/or monitoring items stored thereby (e.g., asmart refrigerator and/or a smart shelving system). The item storagedatabase associated with the user account may include inventory datathat indicates item information associated items currently and/orpreviously stored by the additional device configured to store items(e.g., refrigerator). The computing system may determine that a firstitem is needed for the task, as indicated by the metadata, and using theinventory data, the computing system may determine that the first itemis absent from the additional device. In this particular scenario, thecomputing system may determine text data representing a messageindicating that the first item is unavailable, and may inquire as toif/when the individual would like to purchase the first item. Audio datarepresenting this text data may be generated and sent to the electronicdevice. If the individual says “yes,” as well as potentially indicatinga time that he/she wants the first item to be delivered, the computingsystem may generate purchase data representing a purchase request forthe first item to be bought and delivered for the requested time, andmay send the purchase request to an purchase fulfillment system/service.

In some embodiments, an electronic device, such as those describedpreviously, may be a sound controlled electronic device. A soundcontrolled electronic device, as described herein, may correspond to anydevice capable of being activated in response to detection of a specificword (e.g., a word, a phoneme, a phrase or grouping of words, or anyother type of sound, or any series of temporally related sounds). Forexample, a voice activated electronic device is one type of soundcontrolled electronic device. In response to determining that a wakewordhas been uttered, the voice activated electronic device may, in someembodiments, send the audio data representing a spoken utterancesubsequently following the wakeword to a speech-processing system forprocessing and analyzing the audio data. The speech-processing systemmay then generate and send a response to the voice activated electronicdevice, as well as, or alternatively, communicate with one or moreadditional systems for obtaining content to be rendered by the voiceactivated electronic device, and/or may cause one or more additionalelectronic devices to output content and/or perform a particular action(e.g., turn on a light, preheat an oven, etc.). Additionally, the voiceactivated electronic device may store one or more wakewords within itslocal memory. If a determination is made that audio data received fromthe audio input devices (e.g., microphones) matches the wakeword, thevoice activated electronic device may begin sending audio datarepresenting some or all of the audio captured by the voice activatedelectronic device to the speech-processing system.

Spoken voice commands, in some embodiments, may be prefaced by awakeword, which may also be referred to as a trigger expression, wakeexpression, or activation word. In response to detecting the wakewordbeing uttered, a voice activated electronic device may be configured todetect and interpret any words that subsequently follow the detectedwakeword as actionable inputs or commands. In some embodiments, thevoice activated electronic device may be activated by a phrase orgrouping of words, which the voice activated device may also beconfigured to detect. The voice activated device, therefore, may also beable to detect and interpret any words subsequently following thatphrase or grouping of words.

As used herein, the term “wakeword” may correspond to a “keyword” or“key phrase,” an “activation word” or “activation words,” or a“trigger,” “trigger word,” or “trigger expression.” One exemplarywakeword may be a name, such as the name, “Alexa,” however persons ofordinary skill in the art will recognize that the any word (e.g.,“Amazon”), or series of words (e.g., “Wake Up” or “Hello, Alexa”) mayalternatively be used as the wakeword. Furthermore, the wakeword may beset or programmed by an individual operating a voice activatedelectronic device, and in some embodiments more than one wakeword (e.g.,two or more different wakewords) may be available to activate a voiceactivated electronic device. In yet another embodiment, the trigger thatis used to activate a voice activated device may be any series oftemporally related sounds.

As used herein, the term “utterance” may correspond to a spoken word,statement, or sound. In some embodiments, an utterance may include thewakeword followed by an invocation, such as a request, question, orcommand. In this particular instance, the utterance may begin with thewakeword being spoken, and may end when a last word, phoneme, or soundis spoken. For example, an utterance may correspond to the question,“Alexa—What is the weather currently like?” As another example, anutterance may be, “Alexa—Play my workout music,” or “Alexa—Buy that.”Further still, an utterance, which need not include the wakeword, maybe, “Turn up the volume” or “Call mom.”

Another type of sound controlled electronic device may be a soundactivated electronic device. Such sound activated electronic device mayfunction similarly to voice activated electronic devices, except that,for sound activated electronic devices, the trigger may be a non-verbalsound. For example, the sound of a door opening, an alarm going off,glass breaking, a telephone ringing, or any other sound mayalternatively be used to activate a sound controlled electronic device.In this particular scenario, detection of a non-verbal sound may occurin a substantially similar manner as that of a verbal wakeword for avoice activated electronic device. For example, the sound of a dooropening, when detected, may activate a sound activated electronicdevice, which in turn may activate a burglar alarm.

In some embodiments, the electronic device may additionally, oralternatively, correspond to a manually activated electronic device. Amanually activated electronic device, as described herein, maycorrespond to a device that is capable of being activated in response toa manual input from an individual (e.g., pressing a button, touching aportion of a touch screen, performing an action on a device). Forexample, a tap-to-talk device is one type of manually activated device.Such tap-to-talk devices, for instance, are capable of obtaining andoutputting audio data in response to a button being pressed by anindividual. In some embodiments, a sound controlled electronic device(e.g., a voice activated electronic device and/or a sound activatedelectronic device) may include manual activated functionality such thatan individual may interact with the device using voice (e.g., speechprefaced by a wakeword), sound (e.g., a particular trigger sound),and/or a manual input (e.g., a button press, a touch of a screen, etc.).However, in some embodiments, an electronic device may be configured toinclude a single activation functionality (e.g., only voice activated,or only manually activated).

FIG. 1 is an illustrative diagram of an exemplary system for causing anaction to be performed by a first device based recipe data provided to asecond device, in accordance with various embodiments. In thenon-limiting example embodiment, an individual 2 may speak an utterance4 to a voice activated electronic device 10. For example, individual 2may say, “Alexa, I want to make pizza tonight.” In response todetermining that a wakeword (e.g., “Alexa”) for electronic device 10 wasuttered, electronic device 10 may begin packing and sending audio datarepresenting utterance 4 to a computing system 200 via a network 230.Computing system 200, may perform speech processing to the audio data tocause one or more actions to occur such as, and without limitation,provide a response to utterance 4, cause one or more devices/systems toperform an action, and/or communicate with one or more additionaldevices/systems.

Computing system 200 may, in some embodiments, perform exemplary process150, which may begin at step 152. At step 152, first audio datarepresenting utterance 4 may be received from electronic device 10. Asmentioned previously, the first audio data may be sent to computingsystem 200 in response to electronic device 10 determining that awakeword (e.g., “Alexa,” “Computer,” etc.) had been uttered within anenvironment where electronic device 10 is located. However, in someembodiments, the first audio data may be sent to computing system 200 inresponse to any suitable trigger expression. For example, the triggerexpression may be two or more words, such as a phrase, (e.g., “Hello,Alexa,” “Good Morning,” etc.), a sound trigger (e.g., the sound ofbreaking glass, clapping, etc.), and/or a manual input detected byelectronic device 10. Furthermore, in some embodiments, individual 2 mayinput text data directly to electronic device 10, which may be sent tocomputing system 200, or an additional device/system may send text dataassociated with an input (either audibly spoken, or manually input)originating from electronic device 10 to computing system 200.

At step 154, an account identifier associated with electronic device 10may be received by computing system 200. In some embodiments, theaccount identifier may be sent by electronic device 10 to computingsystem 200 at a substantially same time as the first audio data. Theaccount identifier may, for example, be sent as metadata appended to thefirst audio data, however persons of ordinary skill in the art willrecognize that this is merely exemplary. The account identifier, asdescribed herein, may indicate a particular user account on computingsystem 200 that electronic device 10 is registered to. For example, wheninitially setting up electronic device 10, an individual may indicate tocomputing system 200 that electronic device 10 is to be associated withthat individual's account on computing system 200. That account, whichmay be set up prior to individual 2 “setting up” electronic device 10,may include and store such information as payment information (e.g.,credit card numbers, payment plans), shipping information (e.g., amailing address), user history (e.g., purchase history, interactionhistory, etc.), and/or user preferences. Persons of ordinary skill inthe art will recognize that, in some embodiments, a device identifier(e.g., a media access control (“MAC”) address, IP address, and/or serialnumber) associated with electronic device 10 may alternatively and/oradditionally be sent to computing system 200 from electronic device 10,and the device identifier may be used to determine the accountidentifier.

At step 156, a user account associated with the electronic device may bedetermined. For instance, a user account on computing system 200associated with the account identifier received from electronic device10 may be determined. In some embodiments, the user account may indicatewhich devices/systems are accessible/available for a user associatedwith the user account. The user account may be determined to beassociated with a first device, voice activated electronic device 10, aswell as being associated with a second device, Internet of Things(“IoT”) device 20. For example, IoT device 20 may correspond to a“smart” oven, a “smart” refrigerator, a “smart shelving” system, or anyother suitable type of device, or any combination thereof. Persons ofordinary skill in the art will recognize that the term “smart” asdescribed above may indicate that a particular device including thatdevice's normal functionality and also may be capable of beinginteracted with via network 230. The user account may further storeinformation regarding device identifiers, IP addresses, communicationsinformation, and the like, that may be associated with that particularuser account. Further still, the user account may store informationpersonal to one or more users associated with the user account such as,but not limited to, credit card information, telephone numbers, emailaddresses, mailing addresses, and/or contact lists.

At step 158, first text data representing the first audio data may begenerated. For instance, upon receipt by computing system 200, the firstaudio data may be provided to a speech-processing system associated withcomputing system 200 (as described in greater detail below withreference to FIG. 2B). In one embodiment, an orchestrator component ofthe speech-processing system may receive the first audio data initially,and may provide the first audio data to an automatic speech recognition(“ASR”) system to perform ASR processing. The ASR processing may attemptto match the spoken audio to text using one or more grammar models andlanguage models, such that first text data that represents the firstaudio data may be generated.

At step 160, an intent of the utterance may be determined, where theintent may correspond to a request for a pizza recipe to be provided toelectronic device 10. In some embodiments, upon the first text databeing generated, the orchestrator may provide the first text data to anatural language understanding (“NLU”) system to perform NLU processingto the first text data. The NLU processing may attempt to determine anintent of the utterance based on the first text data and variouslanguage models. For instance, NLU processing may determine that anintent of utterance 4 is for a pizza recipe to be provided to electronicdevice 10 such that individual 2 may prepare a pizza using aninteractive recipe.

At step 162, recipe data representing a pizza recipe may be receivedfrom an instructions database. For instance, a recipe database maycorrespond to one type of instructions database that may be populated byrecipe data associated with various recipes, or more generally lists ofinstructions, that are capable of being followed such that an individualmay perform a specified task or tasks (e.g., preparing a food item,building an object, and the like). Each list of instructions representedby list data representing a list of instructions, which may includeinstruction data representing one or more instructions to beperformed/carried out to complete a particular task. As an illustrativeexample, each recipe stored by the recipe database may be represented byrecipe data, and may include one or more steps to cook a food item. Somelist data (e.g., recipe data) may include instruction metadataassociated with instruction data (e.g., step data representing a step ofthe recipe), which may be capable of being used to assist an individualin performing an action associated with that instruction. For example,step metadata associated with step data corresponding to a particularstep of a recipe may indicate a URL to a video illustrating how toperform that step. As still yet another example, step metadataassociated with a particular step may indicate an action to be performedby another device to complete that step. In one embodiment, recipe datarepresenting a particular recipe may be formatted as an XML file,however persons of ordinary skill in the art will recognize that this ismerely exemplary.

In some embodiments, list data may include item metadata associated withinstruction data representing a particular instruction of the list ofinstructions that indicates one or more items needed to complete theintended task of that list. For example, recipe data may include itemmetadata indicating one or more items (e.g., ingredients) needed to cookthe desired food item associated with that recipe data's recipe (e.g.,flour, oil, chicken, etc.).

In the illustrative embodiment, utterance 4 may correspond to a requestfor a pizza recipe, however persons of ordinary skill in the art willrecognize that this is merely exemplary. For example, an individual mayrequest that an action be performed (e.g., turn on a light), and/orrequest information (e.g., weather information, traffic information,etc.). In the illustrative example, individual 2 may seek to obtain arecipe for making a pizza. Computing system 200 may, beforehand, store apreviously used pizza recipe used by individual 2 with the user account,or if no recipe had been previously used, computing system 200 mayselect a most frequently used recipe by other individuals. Therefore,upon identifying a recipe that corresponds to the intent of utterance 4,computing system 200 may provide recipe data representing the recipe toone or more processors associated with computing system 200 and/or oneor more additional devices/systems.

In some embodiments, a recipe domain associated with computing system200 may, upon receiving the recipe data, may determine display datarepresenting a GUI including the recipe to be generated and sent toelectronic device 10. For example, display data representing a GUI 12may be generated, and the display data may be sent to electronic device10. Electronic device 10 may then be configured to cause GUI 12 to berendered by display screen 212 such that some or all of the recipe isviewable by individual 2 (or any other additional individuals) viadisplay screen 212. For example, GUI 12 may include a text descriptionof a first step of the pizza recipe (e.g., Step 1: Pre-heat oven to 400F).

At step 164, computing system 200 may determine that a first step of thepizza recipe calls for an oven to be pre-heated to a first temperature.In some embodiments, the first step data may include text data that isintended for presentation to a human (e.g., individual 2) and describesa first action for the human to do to prepare the first food item. Forinstance, the recipe data may include first step data representing afirst step (e.g., pre-heat the oven to the first temperature) of thepizza recipe. The first step may be determined, in some embodiments, byone or more processors associated with the recipe functionality ofcomputing system 200. For example, an applet and/or speechlet (e.g., anapplet associated with speech) may be capable of facilitating actionsassociated with a particular functionality (e.g., a first partyapplication, a third party application, etc.) of computing system 200.As an illustrative example, a recipe speechlet may be configured toreceive the recipe data, and may determine the various steps associatedwith the recipe using the recipe data. For instance, the recipespeechlet may determine that the recipe data includes first step datarepresenting a first step of the recipe. The recipe speechlet may,therefore, be capable of identifying that a first step of the recipe isfor an oven to be pre-heated to a first temperature. The specifictemperature may be indicated by metadata associated with the first stepdata that is also included with the received recipe data.

At step 166, computing system 200 may determine that the recipe datafurther includes first step metadata associated with the first stepdata. In one embodiment, the first step metadata may indicate a firsttemperature that an oven is to be pre-heated to. For example, the recipespeechlet may determine that the first step metadata is associated withthe first step data. The first step metadata may indicate, in oneembodiment, an action associated with the first step, a type of deviceassociated with the action, and supplemental information associated withthe action. For example, the first step metadata may indicate that anoven is needed for the first step, that the oven is to be pre-heated toa first temperature, and a particular temperature associated with thefirst temperature (e.g., 400 degrees Fahrenheit).

At step 168, a determination may be made as to whether there is an ovenassociated with the user account. For instance, based on the first stepmetadata indicating that the first step is associated with a particulartype of device performing a first action, the recipe speechlet maydetermine whether the user account is associated with any additionaldevices and, if so, whether those devices are of a same type of deviceas indicated by the first step metadata.

If, at step 168, it is determined that there is a first oven associatedwith the user account, command data representing an instruction to causethe first oven to perform the first action. For instance, the commanddata may represent at least the first step metadata may be generated.For example, the command data may indicate that IoT device 20 is topre-heat to the first temperature. In one embodiment, the recipespeechlet may be configured to generate the command data based on theinformation associated with the first step (e.g., the first step dataand the first step metadata). The command data may indicate the actionto be performed, the device that action is to be performed by, andmetadata associated with the action.

At step 172, second text data representing a first message may bedetermined. For instance, after the command data is sent to IoT device20, the recipe speechlet may be notified that the first step has beencompleted. Therefore, the recipe data may include a message to beprompted to individual 2 in response to receiving notification data fromIoT device 20 indicating that the first step has been completed. Forinstance, the first message may indicate that the recipe for makingpizza has been found, and that the first step has been performed. Atstep 174, second audio data representing the second text data may begenerated. For example, the second text data may be provided to atext-to-speech (“TTS”) system to perform speech synthesis processing,thereby generating audio data. After the second audio data has beengenerated, at step 176, the second audio data may be sent by computingsystem 200 to electronic device 10. In the illustrative embodiment,electronic device 10 may, upon receiving the second audio data, outputmessage 6—“Ok. Here is a pizza recipe. Pre-heating your oven to 400 F.”At a substantially same time, at step 178, the command data may be sentto the first oven. For example, the command data may be sent bycomputing system 200 to IoT 20. In some embodiments, the recipespeechlet may be configured to generate and send the command data,however persons of ordinary skill in the art will recognize that this ismerely exemplary. Furthermore, the command data may be sent prior to, orafter, the audio data is sent to electronic device 10, and theaforementioned is merely exemplary.

If, at step 168, it is determined that there is no oven associated withthe user account, or more generally if it determined that there are nodevices associated with the user account that are of a same type ofdevice as that indicated by the first step metadata, then process 150may proceed to step 180. At step 180, third text data representing asecond message may be determined. The second message may indicate toindividual 2 that he/she needs to perform the action associated with thefirst step. For instance, the second message may indicate to individual2 that he/she needs to pre-heat their oven to the first temperature asthere are no IoT devices 20 that are of a same type of device as theoven indicated as being needed for the first step. For example, thesecond message may be, “Please pre-heat your oven to 400 degreesFahrenheit.” At step 182, third audio data may be generated representingthe third text data. For instance, the third text data may be providedto a text-to-speech system to generate audio data representing the thirdtext data. In some embodiments, step 174 and step 182 may besubstantially similar to each other. At step 184, the third audio datamay be sent to electronic device 10.

In some embodiments, computing system 200 may, upon a step beingcompleted, may be instructed to proceed to the next step of the recipe.For example, after the command data has been sent to IoT device 20, andafter the second audio data has been sent to electronic device 10, therecipe data may cause the recipe speechlet to move to the next step inthe recipe (e.g., step two). Furthermore, the recipe speechlet may beconfigured to continue to monitor the activities of IoT device 20 suchthat, when IoT device 20 completes the pre-heating process, notificationdata may be sent to computing system 200 by IoT device 20 that indicatesthat IoT device 20 has completed the action. In one embodiment, uponreceiving the notification data, computing system 200 may cause therecipe speechlet to generate and send audio data representing anadditional message indicating to individual 2 that the pre-heating hascompleted.

However, if, as described above, the user account does not have a smartoven associated with it, then individual 2 may need to update computingsystem 200 that the step has been performed. For example, the secondmessage may further indicate that individual 2 is to inform computingsystem 200 when the action (e.g., pre-heat an oven) has been completed.Therefore, when individual 2 says, “Alexa, I have pre-heated the oven to400 degrees Fahrenheit,” the recipe speechlet may be configured toproceed to a next step of the recipe.

In the illustrative embodiment, IoT device 20, which may correspond to asmart oven, may be caused to begin heating. However, if an additionalaction is not performed (e.g., putting an item in the oven to cook), thesmart oven may continue to heat, and thus become a potential safetyissue. Therefore, the recipe data may further include instructions thatcause additional command data to be sent to the smart oven in the eventthat one or more other conditions arise. For example, the command datamay further include a time-out feature. When the command data isgenerated and sent to the first oven, the recipe speechlet may begin atimer that indicates when the first oven should finish its task. Forexample, a subsequent step of the pizza recipe may call for a pizza tobe placed in the first oven and cooked for a particular amount of time.Therefore, using the recipe data, the recipe speechlet may determine anapproximate time that the individual performing the tasks to make thepizza should reach that particular step, as well as when that stepshould be finished. Thus, if the smart oven has otherwise not beenturned off prior to the expected end time of the recipe, then the recipedata may include additional command data that causes the smart oven tobe shut off.

In some embodiments, computing system 200 may be configured to monitoradditional IoT devices associated with the user account to determinewhether individual 2 has begun to perform any additional tasks, andtherefore is likely no longer located in a same environment aselectronic device 10 and IoT device 20. For example, the user accountmay also be associated with an automobile and/or a garage openingsystem. If, while the recipe data indicates that the recipe is beingperformed, computing system 200 receives a notification that theautomobile has turned on and/or the garage door opening system hasopened a garage door, electronic device 10 may be prompted to determinewhether individual 2 is still present. For example, a message “Your carjust turned on. Do you still want to cook?”, may be output by electronicdevice 10. If no response to the message is detected within a predefinedamount of time, then computing system 200 may cause additional commanddata to be sent to the first oven that causes the first oven to shutoff. In this way, computing system 200 may prevent the smart oven frommaintaining an elevated temperature while no one is present.

FIGS. 2A and 2B are illustrative diagrams corresponding to the systemarchitecture of FIG. 1 , in accordance with various embodiments. FIG.2A, in one embodiment, illustrates the exemplary system architecture ofFIG. 1 , and in particular, the various components of electronic device10 and an Internet of Things (“IoT”) device 20. For example, electronicdevice 10 may correspond to a voice activated electronic device, and IoTdevice 20 may correspond to a smart oven, as described in greater detailabove. In some embodiments, electronic device 10 may include soundcontrolled functionality, such as one or more voice and/or soundactivated components. In some embodiments, electronic device 10 may beconfigured to communicate with computing system 200 in response todetecting an utterance including a wakeword, which may subsequently befollowed by a request/question/statement. Similarly, electronic device10 may alternatively or additionally include one or more manuallyactivated components for manually activating electronic device 10. Inthis particular scenario, electronic device 10 may also be configured,in one embodiment, to communicate with computing system 200 in responseto a manual input being detected by one or more input mechanisms, suchas a touch screen, a button, and/or a switch, for example.

In a non-limiting embodiment, electronic device 10 may be capable ofbeing activated in response to detecting a specific sound, such as awakeword, as well as, or alternatively, via one or more inputs. Afterdetecting a specific sound (e.g., a wakeword or trigger expression),electronic device 10 may recognize commands (e.g., audible commands,inputs) within captured audio, and may perform one or more actions inresponse to the received commands. Furthermore, electronic device 10 mayalso be configured to perform one or more actions in response todetecting a particular touch, or mechanical, input(s) via electronicdevice 10.

Electronic device 10 may correspond to any suitable type of electronicdevice including, but are not limited to, desktop computers, mobilecomputers (e.g., laptops, ultrabooks), mobile phones, smart phones,tablets, televisions, set top boxes, smart televisions, personal displaydevices, large scale display devices (e.g., billboards, street signs,etc.), personal digital assistants (“PDAs”), gaming consoles and/ordevices, smart furniture, smart household devices (e.g., refrigerators,ovens, microwaves, etc.), smart vehicles (e.g., cars, trucks,motorcycles, etc.), smart transportation devices (e.g., boats, ships,trains, airplanes, etc.), wearable devices (e.g., watches,pins/broaches, headphones, etc.), and/or smart accessories (e.g., lightbulbs, light switches, electrical switches, etc.). In some embodiments,electronic device 10 may be relatively simple or basic in structure suchthat no, or a minimal number of, mechanical input option(s) (e.g.,keyboard, mouse, track pad) or touch input(s) (e.g., touch screen,buttons) are included. For example, electronic device 10 may be able toreceive and output audio, and may include power, processingcapabilities, storage/memory capabilities, and communicationcapabilities. However, in other embodiments, electronic device 10 mayinclude one or more components for receiving mechanical inputs or touchinputs, such as a touch screen and/or one or more buttons.

Electronic device 10, in one embodiment, may include a minimal number ofinput mechanisms (e.g., a power on/off switch) such that functionalityof electronic device 10 may solely or primarily be through audio inputand audio output. For example, electronic device 10 may include, or bein communication with, one or more microphones that listen for awakeword by continually monitoring local audio. In response to thewakeword being detected, electronic device 10 may establish a connectionwith computing system 200, send audio data to computing system 200, andawait/receive a response from computing system 200. In this way,electronic device 10 may provide a voice user interface (“VUI”), suchthat interactions with electronic device 10 may primarily occur viavoice. In some embodiments, however, non-voice/sound activated devicesmay also communicate with computing system 200. For example, in responseto a button or touch screen being pressed, or a button or touch screenbeing pressed and held, a microphone associated with electronic device10 may begin recording local audio, establish a connection withcomputing system 200, send audio data representing the captured audio tocomputing system 200, and await/receive a response, and/or action to beoccur, from computing system 200.

Persons of ordinary skill in the art will recognize that computingsystem 200 may be located within a dedicated computing device orcomputing system, which may or may not be in communication with one ormore additional devices. For instance, computing system 200 may belocated on a remote computing system with which an individual subscribesto a service on. For example, an individual may have a user account oncomputing system 200. However, computing system 200 may also be includedwithin a computing system locally stored or owned by an individual.

Electronic device 10 may include one or more processors 202,storage/memory 204, communications circuitry 206, one or moremicrophones 208 or other audio input devices (e.g., transducers), one ormore speakers 210 or other audio output devices, a display screen 212,and one or more cameras 214 or other image capturing components.However, one or more additional components may be included withinelectronic device 10, and/or one or more components may be omitted. Forexample, electronic device 10 may also include a power supply or a busconnector. As still yet another example, electronic device 10 mayinclude one or more additional input and/or output mechanisms, such asone or more buttons, or one or more switches or knobs. Furthermore,while electronic device 10 may include multiple instances of one or morecomponents, for simplicity only one of each component has been shown.

In some embodiments, electronic device 10 may correspond to a manuallyactivated device, or may include the functionality of a manuallyactivated device. A manually activated device, as described herein, maycorrespond to a device that is capable of being activated in response toa manual input (e.g., pressing a button, touching a portion of a touchscreen, performing an action on a device). For example, a tap-to-talkdevice is one type of manually activated device. Such tap-to-talkdevices, for instance, are capable of obtaining and outputting audiodata in response to a button being pressed.

In one embodiment, electronic device 10 may be in communication with anadditional processing device including one or more of: processor(s) 202,storage/memory 204, communications circuitry 206, audio input(s) 208,audio output(s) 210, display screen 212, and/or camera(s) 214. Forexample, a centralized control device of electronic device 10 mayinclude one or more audio input(s) 208. These audio input(s) 208 mayreceive audio input signals, and electronic device may determine whetheror not the audio input signals indicate that a wakeword was uttered. Ifso, then electronic device may cause audio data representing anutterance including the wakeword, or spoken after the wakeword. To besent to computing system 200.

Processor(s) 202 may include any suitable processing circuitry capableof controlling operations and functionality of electronic device 10, aswell as facilitating communications between various components withinelectronic device 10. In some embodiments, processor(s) 202 may includea central processing unit (“CPU”), a graphic processing unit (“GPU”),one or more microprocessors, a digital signal processor, or any othertype of processor, or any combination thereof. In some embodiments, thefunctionality of processor(s) 202 may be performed by one or morehardware logic components including, but not limited to,field-programmable gate arrays (“FPGA”), application specific integratedcircuits (“ASICs”), application-specific standard products (“ASSPs”),system-on-chip systems (“SOCs”), and/or complex programmable logicdevices (“CPLDs”). Furthermore, each of processor(s) 202 may include itsown local memory, which may store program systems, program data, and/orone or more operating systems. However, processor(s) 202 may run anoperating system (“OS”) for electronic device 10, and/or one or morefirmware applications, media applications, and/or applications residentthereon. In some embodiments, processor(s) 202 may run a local clientscript for reading and rendering content received from one or morewebsites. For example, processor(s) 202 may run a local JavaScriptclient for rendering HTML or XHTML content received from a particularURL accessed by electronic device 10.

Storage/memory 204 may include one or more types of storage mediums suchas any volatile or non-volatile memory, or any removable ornon-removable memory implemented in any suitable manner to store datafor electronic device 10. For example, information may be stored usingcomputer-readable instructions, data structures, and/or program systems.Various types of storage/memory may include, but are not limited to,hard drives, solid state drives, flash memory, permanent memory (e.g.,ROM), electronically erasable programmable read-only memory (“EEPROM”),CD-ROM, digital versatile disk (“DVD”) or other optical storage medium,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, RAID storage systems, or any other storagetype, or any combination thereof. Furthermore, storage/memory 204 may beimplemented as computer-readable storage media (“CRSM”), which may beany available physical media accessible by processor(s) 202 to executeone or more instructions stored within storage/memory 204. In someembodiments, one or more applications (e.g., gaming, music, video,calendars, lists, etc.) may be run by processor(s) 202, and may bestored in memory 204.

In some embodiments, storage/memory 204 may include a media system,which may be configured to facilitate communications between electronicdevices 100 and computing system 200. For example, the media system maystore one or more communications protocols that may be executed byprocessor(s) 202 for facilitating communications for device 100. In someembodiments, a sessions initiation protocol (“SIP”) may be used tofacilitate media transfer between electronic device 10 and one or moreof computing system 200 and another electronic device 10. SIP, forexample, is an application layer protocol that is text based, and mayemploy Real-time Transport Protocol (“RTP”) or Secure Real-timeTransport Protocol (“SRTP”) functions. In particular, PJSIPcommunications functionality may be employed to support audio, video,presence, and messaging communications for electronic device 10. In someembodiments, a Web Real-Time Communications (“WebRTC”) protocols may beemployed by electronic device 10. In a non-limiting embodiment, themedia system may include instructions that indicate which communicationsprotocols to employ for facilitating media transfer between devicesbased on a device type of electronic device 10. For example, ifelectronic device 10 does not include display 212 and/or camera 214,then the media system may indicate that PJSIP should be used, whereas ifelectronic device 10 includes display 212 and/or camera 214 then themedia system may indicate that WebRTC protocols should be used.

In some embodiments, storage/memory 204 may include one or more systemsand/or databases, such as a speech recognition system, a wakeworddatabase, a sound profile database, and a wakeword detection system. Thespeech recognition system may, for example, include an automatic speechrecognition (“ASR”) component that recognizes human speech in detectedaudio. The speech recognition system may also include a natural languageunderstanding (“NLU”) component that determines user intent based on thedetected audio. Also included within the speech recognition system maybe a text-to-speech (“TTS”) component capable of converting text tospeech to be outputted by audio output(s) 210, and/or a speech-to-text(“STT”) component capable of converting received audio signals into textto be sent to speech-processing system for processing.

The wakeword database may be a database stored locally by storage/memory204 of electronic device 10, and may include a list of a currentwakeword for electronic device 10, as well as one or more previouslyused, or alternative, wakewords for voice activated electronic device.In some embodiments, an individual may set or program a wakeword fortheir electronic device 10. The wakeword may be programmed directly onelectronic device 10, or a wakeword or wakewords may be set by theindividual via a local client application that is in communication withcomputing system 200. For example, an individual may use their mobiledevice having the speech-processing system application running thereonto set the wakeword. The specific wakeword may then be communicated fromthe mobile device to computing system 200, which in turn may send/notifyelectronic device 10 of the individual's selection for the wakeword. Theselected activation may then be stored in the wakeword database ofstorage/memory 204. In some embodiments, additional trigger expressionsor permutations of the wakeword may also be stored within storage/memory204. For example, specific trigger expressions or words that indicatethe presence of the wakeword may also be stored within storage/memory204. In some embodiments, audio watermarks, indicating a specific actionor message, may also be stored within storage/memory 204.

In some embodiments, sound profiles for different words, phrases,commands, or audio compositions are also capable of being stored withinstorage/memory 204, such as within a sound profile database. Forexample, a sound profile of audio may be stored within the sound profiledatabase of storage/memory 204 on electronic device 10. In this way, ifa particular sound (e.g., a wakeword or phrase) is detected, acorresponding command or request may be ignored, for example. A soundprofile, for example, may correspond to a frequency and temporaldecomposition of a particular audio file or audio portion of any mediafile, such as an audio fingerprint or spectral representation.

The wakeword detection system may include an expression detector thatanalyzes an audio signal produced by audio input(s) 208 to detect awakeword, which generally may be a predefined word, phrase, or any othersound, or any series of temporally related sounds. Such an expressiondetector may be implemented using keyword spotting technology, as anexample. A keyword spotter is a functional component or algorithm thatevaluates an audio signal to detect the presence of a predefined word orexpression within the audio signal detected by audio input(s) 208.Rather than producing a transcription of words of the speech, a keywordspotter generates a true/false output (e.g., a logical 1/0) to indicatewhether or not the predefined word or expression was represented in theaudio signal. In some embodiments, an expression detector may beconfigured to analyze the audio signal to produce a score indicating alikelihood that the wakeword is represented within the audio signaldetected by audio input(s) 208. The expression detector may then comparethat score to a wakeword threshold to determine whether the wakewordwill be declared as having been spoken.

In some embodiments, a keyword spotter may use simplified ASRtechniques. For example, an expression detector may use a Hidden MarkovModel (“HMM”) recognizer that performs acoustic modeling of the audiosignal and compares the HMM model of the audio signal to one or morereference HMM models that have been created by training for specifictrigger expressions. An HMM model represents a word as a series ofstates. Generally, a portion of an audio signal is analyzed by comparingits HMM model to an HMM model of the trigger expression, yielding afeature score that represents the similarity of the audio signal modelto the trigger expression model.

In practice, an HMM recognizer may produce multiple feature scores,corresponding to different features of the HMM models. An expressiondetector may use a support vector machine (“SVM”) classifier thatreceives the one or more feature scores produced by the HMM recognizer.The SVM classifier produces a confidence score indicating the likelihoodthat an audio signal contains the trigger expression. The confidencescore is compared to a confidence threshold to make a final decisionregarding whether a particular portion of the audio signal represents anutterance of the trigger expression (e.g., wakeword). Upon declaringthat the audio signal represents an utterance of the trigger expression,electronic device 10 may then begin transmitting the audio signal tocomputing system 200 for detecting and responds to subsequent utterancesmade by an individual.

In some embodiments, storage/memory 204 may store voice biometric dataassociated with one or more individuals. For example, an individual thatoperates electronic device 10 may have a registered user account oncomputing system 200 (e.g., within accounts system 268). In someembodiments, electronic device 10 may be associated with a groupaccount, and various individuals may have user accounts that areoperating under the rules and configurations of the group account. As anillustrative example, electronic device 10 may be associated with afirst group account on computing system 200, the first group accountbeing for a family that lives at a household where electronic device 10is located. Each family member may also have a user account that islinked to the first group account (e.g., a parent, a child, etc.), andtherefore each user account may obtain some or all of the rights of thefirst group account. For example, electronic device 10 may have a firstgroup account on computing system 200 registered to a particular familyor group, and each of the parents and children of the family may havetheir own user account registered under the parent's registered account.In one illustrative embodiment, voice biometric data for each individualmay be stored by that individual's corresponding user account. The voicebiometric data, for instance, may correspond to a “voice print” or“voice model” of a particular individual, which may be a graphicalrepresentation of a person's voice including a frequency decompositionof that individual's voice.

Communications circuitry 206 may include any circuitry allowing orenabling one or more components of electronic device 10 to communicatewith one another, and/or with one or more additional devices, servers,and/or systems. For example, communications circuitry 206 may facilitatecommunications between electronic device 10 and computing system 200. Asan illustrative example, audio data representing an utterance (e.g.,utterance 4 of FIG. 1 ) may be transmitted over a network 230, such asthe Internet, to computing system 200 using any number of communicationsprotocols. For example, network(s) 230 may be accessed using TransferControl Protocol and Internet Protocol (“TCP/IP”) (e.g., any of theprotocols used in each of the TCP/IP layers), Hypertext TransferProtocol (“HTTP”), WebRTC, SIP, and wireless application protocol(“WAP”), are some of the various types of protocols that may be used tofacilitate communications between electronic device 10 and computingsystem 200. In some embodiments, electronic device 10 and computingsystem 200 may communicate with one another via a web browser usingHTTP. Various additional communication protocols may be used tofacilitate communications between electronic device 10 and computingsystem 200, including, but not limited to, Wi-Fi (e.g., 802.11protocol), Bluetooth, radio frequency systems (e.g., 900 MHz, 1.4 GHz,and 5.6 GHz communication systems), cellular networks (e.g., GSM, AMPS,GPRS, CDMA, EV-DO, EDGE, 3GSM, DECT, IS-136/TDMA, iDen, LTE or any othersuitable cellular network protocol), infrared, BitTorrent, FTP, RTP,RTSP, SSH, and/or VOIP.

Communications circuitry 206 may use any communications protocol, suchas any of the previously mentioned exemplary communications protocols.In some embodiments, electronic device 10 may include one or moreantennas to facilitate wireless communications with a network usingvarious wireless technologies (e.g., Wi-Fi, Bluetooth, radiofrequency,etc.). In yet another embodiment, electronic device 10 may include oneor more universal serial bus (“USB”) ports, one or more Ethernet orbroadband ports, and/or any other type of hardwire access port so thatcommunications circuitry 206 allows electronic device 10 to communicatewith one or more communications networks.

Electronic device 10 may also include one or more audio inputs 208. Forexample, audio input(s) 208 may correspond to one or more microphonesand/or transducers. Furthermore, one or more audio input devices locatedwithin a separate device may be in communication with electronic device10 to capture sounds for electronic device 10. Audio input(s) 208 may beany suitable component capable of detecting audio signals. For example,audio input(s) 208 may include one or more sensors for generatingelectrical signals and circuitry capable of processing the generatedelectrical signals. In some embodiments, audio input(s) 208 may includemultiple microphones capable of detecting various frequency levels. Asan illustrative example, electronic device 10 may include multiplemicrophones (e.g., four, seven, ten, etc.) placed at various positionsabout electronic device 10 to monitor/capture any audio outputted in theenvironment where electronic device 10 is located. The various audioinputs 208 may include some microphones optimized for distant sounds,while some microphones may be optimized for sounds occurring within aclose range of electronic device 10. In some embodiments, audio input(s)208 may only begin to detect audio signals in response to a manual inputto electronic device 10. For example, a manually activated device maybegin to capture audio data using audio input(s) 208 in response to auser input, such as pressing a button, tapping a touch screen, orproviding any touch input gesture to a touch input component.

Electronic device 10 may include one or more audio output(s) 210. Forexample, audio output(s) 210 may correspond to one or more speakers.Furthermore, electronic device 10 may be in communication with one ormore audio output(s) 210. Audio output(s) 210 may correspond to anysuitable mechanism for outputting audio signals. For example, audiooutput(s) 210 may include one or more speaker units, speaker housings,transducers, arrays of speakers, and/or arrays of transducers that maybe capable of broadcasting audio signals and or audio content to asurrounding area where electronic device 10 may be located. In someembodiments, audio output(s) 210 may include headphones or ear buds,which may be wirelessly wired, or hard-wired, to electronic device 10,that may be capable of broadcasting audio directly to an individual.

In some embodiments, one or more audio input(s) 208 may serve as inputdevices to receive audio inputs. Electronic device 10, in the previouslymentioned embodiment, may then also include one or more audio outputs210 to output audible responses. In this manner, electronic device 10may function solely through speech or audio, without the use or need forany input mechanisms or displays, however this is merely exemplary.

Display screen 212 may correspond to a display device and/or touchscreen, which may be any size and/or shape and may be located at anyportion of electronic device 10. Various types of displays may include,but are not limited to, liquid crystal displays (“LCD”), monochromedisplays, color graphics adapter (“CGA”) displays, enhanced graphicsadapter (“EGA”) displays, variable graphics array (“VGA”) display, orany other type of display, or any combination thereof. Still further, atouch screen may, in some embodiments, correspond to a display deviceincluding capacitive sensing panels capable of recognizing touch inputsthereon. For instance, display screen 212 may correspond to a projectedcapacitive touch (“PCT”), screen include one or more row traces and/ordriving line traces, as well as one or more column traces and/or sensinglines. In some embodiments, display screen 212 may be an optionalcomponent for electronic device 10. For instance, electronic device 10may not include display screen 212. Such devices, sometimes referred toas “headless” devices, may output audio, or may be in communication witha display device for outputting viewable content.

Display screen 212, in one non-limiting embodiment, may include aninsulator portion, such as glass, coated with a transparent conductor,such as indium tin oxide (“InSnO” or “ITO”). In general, one side of thetouch screen display may be coated with a conductive material. A voltagemay be applied to the conductive material portion generating a uniformelectric field. When a conductive object, such as a human finger,stylus, or any other conductive medium, contacts the non-conductiveside, typically an outer surface of display screen 212, a capacitancebetween the object and the conductive material may be formed.Processor(s) 202 may be capable of determining a location of the touchscreen associated with where the capacitance change is detected, and mayregister a touch input as occurring at that location.

In some embodiments, display screen 212 may include multiple layers,such as a top coating layer, a driving line layer, a sensing layer, anda glass substrate layer. As mentioned previously, the glass substratelayer may correspond to an insulator portion, while the top coatinglayer may be coated with one or more conductive materials. The drivingline layer may include a number of driving lines, and the sensing layermay include a number of sensing lines, which are described in greaterdetail below. Persons of ordinary skill in the art will recognize thatone or more additional layers, or spaces between layers, may beincluded. Furthermore, persons of ordinary skill in the art willrecognize that any number of driving lines and sensing lines for drivingthe line layer and the sensing layer, respectively, may be used.

In some embodiments, the driving lines and the sensing lines of thedriving line layer and the sensing line layer, respectively, may form anumber of intersection points, where each intersection functions as itsown capacitor. Each sensing line may be coupled to a source, such that acharge is provided to each sensing line, and changes in capacitance of aparticular driving line and sensing line are detectable thereby. Inresponse to a conductive object being brought proximate, orsubstantially touching an outer surface of the top coating layer, amutual capacitance of a particular capacitor (e.g., an intersectionpoint) may reduce in magnitude. In other words, a voltage drop may bedetected at a location on display screen 212 corresponding to where aconductive object contacted display screen 212.

A change in capacitance may be measured to determine a location on thetouch screen where the object has contacted the surface. For example, ifan individual touches a point on display screen 212, then acorresponding driving line and sensing line that intersect at that pointmay be identified. A location of the point may have one or more pixelsassociated with that location, and therefore one or more actions may beregistered for an item or items that are displayed at that location.Processor(s) 202 of electronic device 10 may be configured to determinewhich pixels are associated with a particular location point, and whichitem or items are also displayed at that pixel location. Furthermore,electronic device 10 may be configured to cause one or more additionalactions to occur to the item or items being displayed on display screen212 based on a temporal duration the touch input, and or if one or moreadditional touch inputs are detected. For example, an object thatcontacted display screen 212 at a first location may be determined, at alater point in time, to contact display screen 212 at a second location.In the illustrative example, an object may have initially contacteddisplay screen 212 at the first location and moved along a particulardriving line to the second location. In this scenario, a same drivingline may have detected a change in capacitance between the twolocations, corresponding to two separate sensing lines.

The number of driving lines and sensing lines, and therefore the numberof intersection points, may directly correlate to a “resolution” of atouch screen. For instance, the greater the number of intersectionpoints (e.g., a greater number of driving lines and sensing lines), thegreater precision of the touch input. For instance, a touch screendisplay screen 212 having 100 driving lines and 100 sensing lines mayhave 100 intersection points, and therefore 100 individual capacitors,while a touch screen display screen 212 having 10 driving lines and 10sensing lines may only have 10 intersection points, and therefore 10individual capacitors. Therefore, a resolution of the touch screenhaving 100 intersection points may be greater than a resolution of thetouch screen having 10 intersection points. In other words, the touchscreen having 100 intersection points may be able to resolve a locationof an object touching the touch screen with greater precision than thetouch screen having 10 intersection points. However, because the drivinglines and sensing lines require a voltage to be applied to them, thismay also mean that there is a larger amount of power drawn by electronicdevice 10, and therefore the fewer driving lines and/or sensing linesused, the smaller the amount of power that is needed to operate thetouch screen display.

In some embodiments, display screen 212 may correspond to ahigh-definition (“HD”) display. For example, display screen 212 maydisplay images and/or videos of 720p, 1080p, 1080i, or any other imageresolution. In these particular scenarios, display screen 212 mayinclude a pixel array configured to display images of one or moreresolutions. For instance, a 720p display may present a 1024 by 768,1280 by 720, or 1366 by 768 image having 786,432; 921,600; or 1,049,088pixels, respectively. Furthermore, a 1080p or 1080i display may presenta 1920 pixel by 1080 pixel image having 2,073,600 pixels. However,persons of ordinary skill in the art will recognize that theaforementioned display ratios and pixel numbers are merely exemplary,and any suitable display resolution or pixel number may be employed fordisplay screen 212, such as non-HD displays, 4K displays, and/or ultradisplays.

In some embodiments, content displayed on display screen 212 may beformatted such that contextual entities and lists are able to beanalyzed by computing system 200 for list management and anaphoraresolution. Context related to the displayed content may includeentities associated with electronic device 10 including, but not limitedto, foreground entities (e.g., lists of items, detail pages), backgroundentities (e.g., songs, audio books), and notification entities. Thecontextual metadata may be structured into context entity slots, listmetadata, and any other additional data available. For example,contextual entity slots may correspond to data used for list resolutionand/or anaphora resolution. The contextual entity slots may be specifiedin domain definitions with corresponding values. The list metadata mayinclude list identifiers, item identifiers for items of a list, andabsolute positions of the list for a particular item (e.g., a first itemof a list, a second item of a list, etc.). Such additional data mayinclude unique identifiers associated with an object, item prices,quantities, and the like.

In some embodiments, electronic device 10 may include one or morecameras 214, corresponding to any suitable image capturing component orcomponents capable of capturing one or more images and/or videos.Camera(s) 214 may, in some embodiments, be configured to capturephotographs, sequences of photographs, rapid shots (e.g., multiplephotographs captured sequentially during a relatively small temporalduration), videos, or any other type of image, or any combinationthereof. In some embodiments, electronic device 10 may include multiplecameras 214, such as one or more front-facing cameras and/or one or morerear facing cameras. Furthermore, camera(s) 214 may be configured torecognize far-field imagery (e.g., objects located at a large distanceaway from electronic device 10) or near-filed imagery (e.g., objectedlocated at a relatively small distance from electronic device 10). Insome embodiments, the camera(s) may be high-definition (“HD”) cameras,capable of obtaining images and/or videos at a substantially largeresolution (e.g., 726p, 1080p, 1080i, etc.). In some embodiments,camera(s) 214 may be optional for electronic device 10. For instance,camera(s) 214 may be external to, and in communication with, electronicdevice 10. For example, an external camera may be capable of capturingimages and/or video, which may then be provided to electronic device 10for viewing and/or processing.

Persons of ordinary skill in the art will recognize that, in someembodiments, display screen 212 and/or camera(s) 214 may be optional forelectronic device 10. For instance, electronic device 10 may functionusing audio inputs, and outputting audio in response or causing one ormore actions to occur in response, and therefore display screen 212and/or camera(s) 214 may not be included. Furthermore, in someembodiments, electronic device 10 may not include display screen 212and/or camera(s) 214, but instead may be in communication with displayscreen 212 and/or camera(s) 214. For example, electronic device 10 maybe connected to a display screen via a Wi-Fi (e.g., 802.11 protocol)connection such that visual content sent to electronic device 10 may besent to the display screen, and output thereby.

In one exemplary embodiment, electronic device 10 may include anadditional input/output (“I/O”) interface. For example, electronicdevice 10 may include one or more input components capable of receivinguser inputs. Various types of input components may include, but are notlimited to, keyboards, buttons, switches, a mouse, joysticks, or anexternal controller may be used as an input mechanism for the I/Ointerface. In some embodiments, the output portion of the I/O interfaceof electronic device 10 may include one or more lights, light emittingdiodes (“LEDs”), or other visual indicator(s). Persons of ordinary skillin the art will recognize that, in some embodiments, one or morefeatures of the output portion of the I/O interface may be included in apurely voice activated version of electronic device 10. For example, oneor more LED lights may be included on electronic device 10 such that,when audio input(s) 208 receive audio, the one or more LED lights becomeilluminated signifying that audio has been received by electronic device10. In some embodiments, one or more vibrating mechanisms or otherhaptic features may be included with electronic device 10 to provide ahaptic response to an individual.

In some embodiments, electronic device 10 may include radio-frequencyidentification (“RFID”) functionality and/or near field communication(“NFC”) functionality. Furthermore, in some embodiments, electronicdevice 10 may include one or more infrared (“IR”) sensors and one ormore IR emitters. The IR sensors/emitters may be used to determine depthinformation. For example, in one embodiment, a distance of an individualfrom electronic device 10 may be determined using the one or more IRsensors/emitters. Depth determination may be performed using any depthdetermination technique. In some embodiments, a distance between anindividual and electronic device 10 may be employed as a basis forpresenting content with varying density using display screen 212. Insome embodiments, electronic device 10 may include beaconingfunctionality that allows electronic device 10 to recognize when one ormore devices are located nearby. For example, electronic device 10 mayinclude beaconing detection functionality that generates a beaconingsignal, and attempts to determine whether any devices nearby are capableof detecting that signal. If so, then that may indicate that aparticular device is proximate to electronic device 10.

In some embodiments, contextual metadata may be obtained by computervision analysis of an object detected by camera(s) 214. For example, inresponse to speaking the utterance, “Buy this,” electronic device 10 maycause camera(s) 214 to capture an image. That image may be analyzed todetermine what the object is, and the contextual metadata associatedwith that objects identify may be determined. For instance, if anindividual is holding up a bottle of ketchup, then the computer visionanalysis may be able to determine a product name, bar code, and/or anyother attribute about the bottle of ketchup from the captured image, andmay populate a contextual metadata structure indicating the determinedvalues (e.g., ‘Item Name’ slot: “Ketchup”).

IoT device 20, in some embodiments, may include processor(s) 202,storage/memory 204, communications circuitry 206, and one or morefunction(s) 216. Optionally, IoT device 20 may also include audioinput(s) 208, audio output(s) 210, display screen 212, and/or camera(s)214. IoT device 20 may be capable of being controlled eitherindividually (e.g., by a user directly) or via electronic device 10. Forexample, an individual may cause IoT device 20 to perform an action byproviding an input to IoT device 20 (e.g., voice input, sound input,manual input, etc.). As another example, an individual may speak anutterance, cause a sound to be produced, and/or provide a manual inputto electronic device 10 that, in turn, caused IoT device 20 to performone or more desired actions.

IoT device 20 may correspond to any suitable type of electronic deviceincluding, but are not limited to, desktop computers, mobile computers(e.g., laptops, ultrabooks), mobile phones, smart phones, tablets,televisions, set top boxes, smart televisions, personal display devices,large scale display devices (e.g., billboards, street signs, etc.),personal digital assistants (“PDAs”), gaming consoles and/or devices,smart furniture, smart household devices (e.g., refrigerators, ovens,microwaves, etc.), smart vehicles (e.g., cars, trucks, motorcycles,etc.), smart transportation devices (e.g., boats, ships, trains,airplanes, etc.), wearable devices (e.g., watches, pins/broaches,headphones, etc.), and/or smart accessories (e.g., light bulbs, lightswitches, electrical switches, etc.). In some embodiments, IoT device 20may be substantially similar to electronic device 10, with the exceptionthat IoT device 20 may be capable of performing one or more additionalfunctions using function(s) component(s) 216.

Function(s) component(s) 216 may correspond to any suitable componentsassociated with IoT device 20 that causes IoT device 20 to perform oneor more specified functions. As an illustrative example, IoT device 20may correspond to a smart oven, as described above for FIG. 1 . In thisparticular scenario, function(s) component(s) 216 may correspond to oneor more heating functions (e.g., an oven that heats, stove top,microwave features, etc.) that allow smart oven 20 to perform thedesired functions of an oven. As another example, IoT device 20 maycorrespond to a laundry machine. Therefore, in this particular scenario,function(s) component(s) 216 may correspond to one or more clotheswashing features such that the laundry machine may wash one or moreitems. Persons of ordinary skill in the art will recognize that IoTdevice 20 may correspond to any suitable device, and the aforementionedare merely exemplary.

FIG. 2B, in one embodiment, illustrates the exemplary systemarchitecture of FIG. 1 , and in particular, computing system 200.Computing system 200 may include various components and modulesincluding, but not limited to, speaker identification system 248,automatic speech recognition (“ASR”) system 258, natural languageunderstanding (“NLU”) system 260, applications/skills system 262,text-to-speech (“TTS”) module 264, and accounts system 268. In someembodiments, computing system 200 may also include an orchestratorsystem 250 capable of orchestrating one or more processes to beperformed by one or more of speaker identification system 248, ASRsystem 258, NLU system 260, applications/skills system 262, TTS system264, and/or accounts system 268, as well as one or more additionalcomponents, devices, and/or systems associated therewith. Computingsystem 200 may also include computer readable media, including, but notlimited to, flash memory, random access memory (“RAM”), and/or read-onlymemory (“ROM”). Computing system 200 may also include various modulesthat store software, hardware, logic, instructions, and/or commands forcomputing system 200.

ASR system 258 may be configured to recognize human speech in detectedaudio, such as audio captured by electronic device 10, which may then betransmitted to computing system 200. ASR system 258 may include, in oneembodiment, one or more processor(s) 252, storage/memory 254, andcommunications circuitry 256. Processor(s) 252, storage/memory 254, andcommunications circuitry 256 may, in some embodiments, be substantiallysimilar to processor(s) 202, storage/memory 204, and communicationscircuitry 206, which are described in greater detail above, and theaforementioned descriptions may apply. Furthermore, in some embodiments,ASR system 258 may include speech-to-text (“STT”) system 266. STT system266 may employ various speech-to-text techniques. However, techniquesfor transcribing speech into text are well known in the art and need notbe described in further detail herein, and any suitable computerimplemented speech to text technique may be used to convert the receivedaudio signal(s) into text, such as SOFTSOUND speech processingtechnologies available from the Autonomy Corporation, which isheadquartered in Cambridge, England, United Kingdom.

ASR system 258 may include an expression detector that analyzes audiosignals received by computing system 200, such as the expressiondetector mentioned above with regards to electronic device 10. Such anexpression detector may be implemented using keyword spottingtechnology, as an example. A keyword spotter is a functional componentor algorithm that evaluates an audio signal to detect the presence of apredefined word or expression, such as a passphrase or other sound data,within the audio signals. Rather than producing a transcription of wordsof the speech, a keyword spotter generates a true/false output (e.g., alogical 1/0) to indicate whether or not the predefined word orexpression was represented in the audio signal. In some embodiments, anexpression detector may be configured to analyze the audio signal toproduce a score indicating a likelihood that a particular word or phrase(e.g., a wakeword), is represented within the audio signal. Theexpression detector may then compare that score to a threshold value todetermine whether that word or phrase will be declared as having beenspoken.

For instance, ASR system 258 may transcribe received audio data intotext data representing the words of the speech contained in the audiodata using STT system 266. The text data may then be used by othercomponents for various purposes, such as executing system commands,inputting data, etc. ASR system 258 may then interpret an utterancebased on the similarity between the utterance and pre-establishedlanguage models stored in an ASR model knowledge base of storage/memory254. For example, the input audio data may be compared with models forsounds (e.g., sub-word units or phonemes) and sequences of sounds toidentify words that match the sequence of sounds spoken in the utteranceof the audio data.

The different ways a spoken utterance may be interpreted (i.e., thedifferent hypotheses) may each be assigned a probability or a confidencescore representing a likelihood that a particular set of words matchesthose spoken in the utterance. The confidence score may be based on anumber of factors including, for example, the similarity of the sound inthe utterance to models for language sounds (e.g., an acoustic model),and the likelihood that a particular word which matches the sounds wouldbe included in the sentence at the specific location (e.g., using alanguage or grammar model). Thus each potential textual interpretationof the spoken utterance (hypothesis) is associated with a confidencescore. Based on the considered factors and the assigned confidencescore, ASR system 258 may output the most likely text recognized in theaudio data. ASR system 258 may also output multiple hypotheses in theform of a lattice or an N-best list with each hypothesis correspondingto a confidence score or other score (such as probability scores, etc.).

ASR system 258 may further attempt to match received feature vectors tolanguage phonemes and words as known in acoustic models and languagemodels stored within storage/memory 254 of ASR system 258. Recognitionscores may be determined for the feature vectors based on acousticinformation and language information. The acoustic information may beused to calculate an acoustic score representing a likelihood that theintended sound represented by a group of feature vectors matches alanguage phoneme. The language information may be used to adjust theacoustic score by considering what sounds and/or words are used incontext with each other, thereby improving the likelihood that thespeech recognition process will output speech results that make sensegrammatically. The specific models used may be general models or may bemodels corresponding to a particular domain, such as music, applicationenablement/disablement, shopping, etc.

ASR system 258 may generate results in the form of a single textualrepresentation of the speech, an N-best list including multiplehypotheses and respective scores, and/or lattice, for example, which maybe sent to NLU system 260 for processing, such as conversion of the textinto commands for execution, either by electronic device 10, computingsystem 200, or by another device, such as a separate device or servercapable of performing one or more additional functionalities thereon(e.g., a television capable of outputting video content).

NLU system 260 may be configured such that it determines an intent of anutterance based on the received audio data. NLU system 260 may determineone or more domains, which may also be referred to as categories, thatmay be capable of handling the intent of the utterance. For example, anutterance, “Play this,” may be identified by a Music domain, an E-Bookdomain, and a Video domain as possibly being able to handle thecorresponding request. For instance, NLU system 260 may identify thatthe word “Play” may be a recognized intent of each of the aforementioneddomains (as well, possibly, other domains). In some embodiments, todetermining an utterance's intent, NLU system 260 may communicate withapplications/skills system 262 to cause one or more specificfunctionalities to be accessible, perform one or more tasks, and/orretrieve an appropriate response or response information. NLU system 260may include processor(s) 252, storage/memory 254, and communicationscircuitry 256 which, in one embodiment, may be substantially similar toprocessor(s) 202, storage/memory 204, and communications circuitry 206of electronic device 200, and the previous description may apply.

NLU system 260 may include a named entity recognition (“NER”) system272, which may be used to identify portions of text that correspond to anamed entity recognizable by NLU system 260. A downstream process callednamed entity resolution may be configured to link a portion of text toan actual specific known entity. To perform named entity resolution, thesystem may utilize gazetteer information stored in an entity librarystorage. The gazetteer information may be used for entity resolution,for example matching ASR results with different entities (such as songtitles, contact names, etc.). Gazetteers may be linked to a user accountor profile of users accounts module 268, certain domains (e.g., music orshopping), or may be organized in a variety of other ways.

Generally, NLU system 260 takes textual input and attempts to make asemantic interpretation of the text. That is, NLU system 260 may beconfigured to determine a meaning of text based on the individual wordsand then implements that meaning. In some embodiments, NLU system 260may interpret a text string to derive an intent or a desired action ofthe utterance (e.g., utterance 4) as well as the pertinent pieces ofinformation in the text that allow an action to be completed. Forexample, if a spoken utterance is processed by ASR system 258 andoutputs the text, “call mom,” NLU system 260 may determine that anintent of the utterance is to activate a telephone, or telephonefunctionality, and to initiate a call with a contact matching the entity“mom”. In some embodiments, NLU system 260 may process several textualinputs related to the same utterance. For example, if ASR system 258outputs N text segments (as part of an N-best list), then NLU system 260may process all N outputs.

As will be discussed further below, NLU system 260 may be configured toparse and tag annotate text. For example, for the text “call mom,”“call” may be tagged as a command (e.g., a command to execute a phonecall), and “mom” may be tagged as a specific entity and target of thecommand (e.g., a telephone number for the entity corresponding to “mom”stored in a contact list). Further, NLU system 260 may be used toprovide answer data in response to queries, for example using aknowledge base stored within storage/memory 254 of NLU system 260 and/orstorage/memory of computing system 200.

To correctly perform natural language understanding processing of speechinput, NLU system 260 may be configured to determine a domain of anutterance. By determining the domain, NLU system 260 may narrow downwhich services and functionalities offered by an endpoint device (e.g.,electronic device 10, computing system 200, or any other electronicdevice or system) may be relevant. For example, an endpoint device mayoffer services relating to interactions with a telephone service, acontact list service, a calendar/scheduling service, a music playerservice, etc. As another example, an endpoint device may enable certainservices or functionalities for an individual having a user account ofcomputing system 200. Words in a single text query may implicate morethan one service, and some services may be functionally linked (e.g.,both a telephone service and a calendar service may utilize data fromthe contact list).

NER system 272 may be configured to receive a query in the form of oneor more results from ASR system 258. NER system 272 may then attempt toidentify relevant grammars and lexical information that may be used toconstrue meaning of the one or more results. To do so, NER system 272may begin by identifying potential domains that may relate to thereceived query. NLU system 260, may include a databases of deviceswithin storage/memory 254 of NLU system 260 that may be used to identifydomains associated with specific devices. For example, electronic device10 may be associated with domains for music, telephone functionality,calendar information, contact lists, and/or device-specificcommunications. In addition, NLU system 260 may store an entity libraryincluding database entries for specific services available on a specificdevice or devices, either indexed by that device's identificationnumber, a customer identification number, a household identificationnumber, and/or using any other suitable indicator.

In one non-limiting embodiment, a domain may represent a discrete set ofactivities, services, and/or functionalities that have a common theme,such as “shopping,” “music,” or “videos.” As such, each domain may beassociated with a particular language model and/or grammar database, aparticular set of intents/actions, and a particular personalizedlexicon. In some embodiments, each user account may have its ownlanguage model including words, phrases, sentence structures, responsetemplates, and the like, configured for the user account. Each gazetteermay include domain-indexed lexical information associated with aparticular user account of accounts system 268 and/or electronic device10. For example, a first gazetteer may include first domain-indexlexical information. A user's music-domain lexical information mightinclude album titles, artist names, and song names, for example, whereasa user's contact-list lexical information might include the names ofcontacts. Since every user's music collection and contact list ispresumably different, this personalized information improves entityresolution.

In some embodiments, NLU system 260 may be configured to apply therules, models, and information applicable to each identified domain. Forexample, if a query potentially implicates both communications andmusic, the query may, substantially in parallel, the natural languageunderstanding processing may use the grammar models and lexicalinformation for communications, and may also use the grammar models andlexical information for music. The responses based on the query producedby each set of models is scored, with the overall highest ranked resultfrom all applied domains is ordinarily selected to be the correctresult.

An intent classification (“IC”) system 274 may parse the query todetermine an intent or intents for each identified domain, where theintent corresponds to the action to be performed that is responsive tothe query. Each domain is associated with a database of words linked tointents. For example, a music intent database of a music domain may linkwords and phrases such as “play,” to a play music intent, “stop,” to astop playing music intent, and “mute” to a mute volume intent. IC system276 may be configured to identify potential intents for each domain bycomparing words in the query to the words and phrases in that domain'sintents database. Traditionally, the determination of an intent by ICsystem 274 is performed using a set of rules or templates that areprocessed against the incoming text to identify a matching intent.

In order to generate a particular interpreted response, NER system 272applies the grammar models and lexical information associated with therespective domain to actually recognize one or more entities in the textof the query. Each grammar model includes the names of entities (i.e.,nouns) commonly found in speech about the particular domain (i.e.,generic terms), whereas the lexical information from the gazetteer ispersonalized to the user(s) and/or the device. For instance, a grammarmodel associated with the shopping domain may include a database ofwords commonly used when people discuss shopping.

The intents identified by IC system 274 may be linked to domain-specificgrammar frameworks having “slots” or “fields” to be filled. Each slot orfield may correspond to a portion of the query text that the systembelieves corresponds to an entity. For example, if “play music” is anidentified intent, a grammar framework or frameworks may correspond tosentence structures such as “Play {Artist Name},” “Play {Album Name},”“Play {Song name},” “Play {Song name} by {Artist Name},” etc. However,to make resolution more flexible, these frameworks would ordinarily notbe structured as sentences, but rather based on associating slots withgrammatical tags. As another example, if “Play ‘Song 1’” is anidentified intent, a grammar framework may correspond to sentencestructures such as “Play {Song 1}.”

NER system 272 may parse the query to identify words as subject, object,verb, preposition, etc., based on grammar rules and/or models, prior toresolving named entities. The identified verb may be used by IC module274 to identify intent, which is then used by NER system 272 to identifyframeworks. A framework for an intent of “play” may specify a list ofslots/fields/placeholders applicable to place the identified “object”such as, for example, {Artist Name}, {Album Name}, {Song Name},{Application Name}, {Anaphoric Term}, and any object modifier (e.g., aprepositional phrase). NER system 272 may then search the correspondingfields in the domain-specific and personalized lexicon(s), attempting tomatch words and phrases in the query, which are tagged as a grammaticalobject or object modifier, with those identified in the database(s).

This process may include semantic tagging, which is the labeling of aword or a combination of words according to their type/semantic meaning.Parsing may be performed using heuristic grammar rules, or an NER modelmay be constructed using techniques such as hidden Markov models,maximum entropy models, log linear models, conditional random fields(“CRF”), and the like.

For instance, a query of “Play ‘Song 1’ by ‘Artist 1’” might be parsedand tagged as {Verb}: “Play,” {Object}: “Song 1,” {Object Preposition}:“by,” and {Object Modifier}: “Artist 1.” At this point in the process,“Play” may be identified as a verb based on a word database associatedwith the music domain, which IC module 276 may determine corresponds tothe “play music” intent. No determination has been made as to themeaning of “Song 1” and “Artist 1,” but based on grammar rules andmodels, it may be determined that the text of these phrases relate tothe grammatical object (i.e., entity) of the query.

The frameworks linked to the intent may then be used to determine whatdatabase fields should be searched to determine the meaning of thesephrases, such as searching a user's gazette for similarity with theframework slots. So a framework for a “play music” intent might attemptto resolve the identified object for {Artist Name}, {Album Name}, {SongName}, and {Application Name}, {Anaphoric Term} and another frameworkfor the same intent might attempt to resolve the object modifier basedon {Artist Name}, and resolve the object based on {Album Name} and {SongName} linked to the identified {Artist Name}. If the search of thegazetteer does not resolve a slot/field using gazetteer information, NERsystem 272 may search the database of generic words associated with theparticular domain. So for instance, if the query was “play songs by‘Artist 1,’” after failing to determine an album name or song namecalled “songs” by “Artist 1,” NER system 272 may search the domainvocabulary for the word “songs.” For example, use of the object “songs”may correspond to some or all of the songs associated with a particularartist (e.g., “Artist 1”). In the alternative, generic words may bechecked before the gazetteer information, or both may be tried,potentially producing two different results.

The results of the natural language understanding processing may betagged to attribute meaning to the query. So, for instance, “Play ‘Song1’ by ‘Artist 1’” might produce a result of: {Domain}: “Music,”{Intent}: “Play Music,” {Artist Name}: “Artist 1,” {Media Type}: “Song,”and {Song Name}: “Song 1.” As another example, “Play songs by ‘Artist1’” might produce: {Domain}: “Music,” {Intent}: “Play Music,” {ArtistName}: “Artist 1,” and {Media Type}: Song. Still further, “Add this tomy cart’” might produce a result of: {Domain} Shopping, {Intent} “Additem to,” {Anaphoric Term}: “this,” and {List Type} “cart.”

The output from NLU system 260 (which may include tagged text, commands,etc.) may then be sent to orchestrator 250 and/or a command processor,which may be located on, or may be in communication with, computingsystem 200. The destination command processor may be determined based onthe output of NLU system 260. For example, if NLU system 260 outputsinclude a command to play music, the destination command processor maybe a music playing application, such as one located on electronic device10 or in a music playing application, configured to execute a musicplaying command to cause a particular audio file to output. If theoutput of NLU system 260, however, includes a search request, thedestination command processor may include a search engine processor,such as one located on a search server, configured to execute a searchcommand. If the output of NLU system 260 includes a request forenablement of an application's functionalities for a particular useraccount, then the language model for the user account may be updated toinclude one or more invocations, utterance frameworks, responseframeworks, and values for the frameworks, for a particular application.

In some embodiments, NLU system 260 may also include an entityresolution module 278, which allows NLU system 260 to query each domainof NLU system 260 to determine which domain or domains believe that theycan handle a given request. Upon performing the query, each domain ofdomain system 272 may return a “score” indicating a likelihood that theycan handle the application in question. For instance, various domainsmay return scores of LOW, MEDIUM, or HIGH, indicating the likelihoodthat they can handle the particular application. Entity recognitionmodule 278 may return application names or titles for the request, whichmay be based on relevance scores that estimate a confidence that aparticular application may be best used for the request. As anillustrative example, if the utterance is, “Play my music,” NLU system260 may determine, using entity resolution module 278, which domains,and thus applications, can likely handle this request, and may selectthe domain and/or application having the highest confidence score asbeing the domain and/or application able to handle the request.Furthermore, as described in greater detail below with reference to FIG.2B, NLU system 260 may further include a slot filler module 276.

In some embodiments, NLU system 260 may include a contextual resolutionsystem 270, which may be a system configured to identify entities forassisting natural language understanding processing using contextualmetadata corresponding to content being displayed by display screen 212.Contextual resolution system 270 may receive contextual metadata fromorchestrator 250 (e.g., that may be generated and sent from a particulardomain of functionalities 262) and/or from electronic device 10. Thecontextual metadata my include information associated with theparticular content currently being rendered by a requesting device. Insome embodiments, the contextual metadata may be obtained in response toreceiving audio data. For example, in response to receiving audio datarepresenting utterance 4, voice activated electronic device 10 may senda notification to computing system 200 that indicates that content iscurrently being rendered by display screen 212.

In one embodiment, orchestrator 250 may be configured to receive theaudio data, and may determine that the notification has also beenreceived. Orchestrator 250 may determine whether or not the notificationindicates that there is (or was) content displayed by display screen 212at the time that the utterance was spoken (e.g., when the wakeword wasuttered). However, persons of ordinary skill in the art will recognizethat the determination of whether content is being displayed by displayscreen 212 may correspond to a time just prior to the wakeword beinguttered, when the wakeword was uttered, after the wakeword was uttered,during the utterance being spoken, after the utterance was spoken,and/or after the audio data is sent to computing system 200, and theaforementioned is merely exemplary. The notification may correspond to aflag, such as a logical 1/0, where if the flag has a value of 1, thenthis may indicate that content is being rendered by electronic device10, whereas if the flag has a value of 0, then this may indicate thatnot content is being rendered by electronic device 10.

Upon determining that electronic device 10 is rendering content (e.g.,receiving a notification indicating that content is being displayed bydisplay screen 212), orchestrator 250 may generate and send a request tothe various domains associated with applications/skills system 262 andNLU system 260. The request may inquire as to which domain, or domains,are currently providing electronic device 10 with content, such as thecontent being rendered. Upon determining that domain, or domains, thatare providing content, orchestrator 250 may request that the identifieddomain(s) generate and send contextual metadata representing textcorresponding to the displayed content. After receiving the contextualmetadata from the corresponding domain(s), orchestrator 250 may providethe contextual metadata to contextual resolution system 270, orotherwise cause the contextual metadata to be sent to NLU system 260.

In some embodiments, the various domains may include formatting logic(e.g., speechlets) that may format the contextual metadata. Thecontextual metadata may be structured, in response to the formattinglogic applying its formatting rules, such that attributes of the contentare arranged in contextual slots including values associated with thoseslots. For instance, the contextual metadata may also be referred to, ormay generally correspond to, entity data representing content beingdisplayed by display screen 212 an utterance is spoken. For example,single item or detail web page of an item, such as a coffee maker, mayinclude entity data organized into contextual slots such as {Item Name},{List Position}, and {Item Identifier}. Values from the particular webpage being displayed currently by display screen 212 may include: {ItemName}: “Coffee Maker,” {List Position}: “1,” and {Item Identifier}:“abcd1234.” Different types of content may include different contextualslots. For example, lists may include {List Types}, {List Position},{List Name}, etc.

In some embodiments, contextual resolution system 270 may determinewhether any slots/fields from intent resolution processing by NLU system260 substantially match any slots/fields from the contextual metadatareceived. For instance, the entity data that is received from the domainmay include similar entities (e.g. slots) as the entities associatedwith the intent identified by NLU system 260. This may include havingcontextual slots from the contextual metadata being associated withsimilar attributes as those of slots from intent resolution by NLUsystem 260. As an example, the “Play Music” intent may include slots forapplication data—{Application Data Slot}, song name—{Song Name}, albumname—{Album Name}, artist name—{Artist Name}, genre name—{Genre Name},playlist name {Playlist Name}, media type—{Media Type}, sort type—{SortType}, play mode—{Play Mode}, service name—{Service Name},anaphor—{Anaphor}, list position—{List Position}, recommendtrigger—{Recommended Trigger}, and similar trigger—{Similar Trigger}.From NLU system 260, the text data may be used to identify some or allof the values for these slots. For example, if the utterance is, “Play‘Song 1’,” then {Song Name} may have a value “Song 1.” However, theremaining slots may remain unfilled. The contextual metadata mayindicate information regarding a GUI displayed by display screen 212,where the GUI includes a detailed page associated with a song (e.g.,“Song 1”) or be a list including one item, the song. For example, thecontextual metadata may include a song name contextual slot, {SongName}, with a filled value being “Song 1,” as well as an album namecontextual slot, {Album Name}, having a filled value “Album 1.” In thisparticular scenario, the album name contextual slot {Album Name} may beprovided to NLU system 260.

Contextual resolution system 270, in one embodiment, may be configuredto determine a heuristics score that indicates a number of matchingentities (e.g., matching slots) between the entity data/contextualmetadata and the declared slots for the identified intent. For example,using the previous example, the contextual metadata may include suchslots as {Song Name}, {Album Name}, and/or {Artist Name}, amongst otherslots. Therefore, in this particular scenario, the heuristics score maybe three, indicating that there are three slots corresponding to similarentities for both the declared slots from IC component 274 andcontextual resolution system 270. If the heuristics score is greaterthan zero, then NLU system 260 may be configured to generate a selectedcontext file that may be included with the output data from NLU system260, which orchestrator 250 may provide back to an application, orapplications, of applications/skills system 262 to perform, or attemptto perform, one or more actions.

Applications/skills system 262 may, for example, correspond to variousaction specific applications, which are capable of processing varioustask specific actions and/or performing various functionalities.Applications/skills system 262 may further correspond to first partyapplications and/or third party applications capable of performingvarious tasks or actions, or performing various functionalities. Forexample, based on the context of the audio received from electronicdevice 10, computing system 200 may use a certain application togenerate a response, or to obtain response information, which in turnmay be communicated back to electronic device 10 and/or to anotherelectronic device (e.g., a television). Applications/skills system 262may also include processor(s) 252, storage/memory 254, andcommunications circuitry 256.

In some embodiments, applications/skills system 262 may include aninstructions database 280. Instructions database 280 may be accessed bycomputing system 200 in response to NLU system 260 determining that arequest corresponds to a task to be performed using one or more lists ofinstructions stored thereby. For example, if an individual requests apizza recipe, then NLU system 260 may recognize that the intent of therequest is for a pizza recipe to be provided to the requesting device.The domain associated with this intent may cause computing system 200 toaccess instruction database 280 to obtain a pizza recipe.

Instructions database 280 may store various lists of instructions oftasks capable of being completed. For instance, lists 282A-N may bestored within instructions database 280. Each list of instructions maydiffer from one another, however some lists may be similar. For example,two or more pizza recipes may be included by lists 282A-N. Lists 282A-Nmay, in some embodiments, be written in an extensible markup language(“XML”) such that various instructions may be written by a variety ofdifferent individuals, companies, and/or systems. For instance,instruction data representing a list of instructions may be stored foreach list of lists 282A-N. In some embodiments, additional data, such asdisplay data representing content to be rendered may be stored for eachlist of instructions 282A-N. For example, the content may correspond toa graphical user interface capable of displaying some or all of arecipe. In response to a particular list being selected for use, thedisplay data may be provided to the corresponding speechlet, which maygenerate and send the display data to the requesting device.

In some embodiments, instruction database 280 may further store metadata284A-N. For instance, first list of instructions 282A may be associatedwith first metadata 284A, which may correspond to metadata related toone or more steps included within first list 282A. As an illustrativeexample, metadata 284A may include URLs to videos instructing how aparticular step of first list 282A may be performed. As anotherillustrative example, metadata 284A may include command data correspondsto actions that may be performed by one or more devices to accomplish atask associated with first list 282A. Instruction database 280 may alsoinclude timers 292A-N. Times 292A-N may be monitored by computing system200. For example, if first list 282A includes a step that says to bakean item for a certain amount of time, the speechlet associated with therecipe functionality may cause timer 292A to start a timer for thatamount of time. At the conclusion of the amount of timer, timer 292A maynotify the speechlet that the timer has ended, and the speechlet maycause one or more actions (e.g., prompting the user to remove the itemfrom an oven, sounding an alarm, outputting a message, etc.) to occur.In some embodiments, two or more of timers 292A-N may operate at asubstantially same time, albeit unrelated to one another.

A particular list of instructions may be selected based on the variouscriteria identified by the request. For instance, an instructions domainof NLU system 260 may include slots that may be capable of specifying atask to be performed. In some embodiments, the instructions domain maybe associated with such slots as a “Item To Be Made” slot, an“Instruction Builder” slot, and/or a “Task To Be Performed” slot. The“Item To Be Made” slot may specify a specific item that is to be made.For example, if an individual says, “I want to make pizza,” the NERsystem 272 and IC system 274 may parse and tag the word “pizza” to beattributed to the “Item To Be Made” slot. The “Instruction Builder” slotmay specify a particular instruction builder with which instructionswere created by. For example, if the individual said, “I want to makelasagna using ‘Chef 1's’ recipe,” the “Item To Be Made” slot may beattributed with the value “lasagna,” while the “Instruction Builder”slot may be attributed with the value “Chef 1.” The “Task To BePerformed” slot, similar, may specify a specific task to be performed.For example, the utterance, “I want to make pizza,” may have the “TaskTo Be Performed” slot attributed with the value “make,” indicating thatan item is to be made via a list of instructions.

In some embodiments, applications/skills system 262 may further includea storage database 296. Storage database 296 may store inventory datarepresenting information associated with one or more storage devices,such as devices 286A-N, 288A-N, and/or 294A-N. A user account mayinclude a smart storage unit/device/system, which may include sensors,memory, processors, and communications circuitry, amongst otherfeatures, capable of determining whether an object is stored by thatsmart storage device, an amount (e.g., a weight) of the object storedthereby, and a location of that item within/about the smart storagedevice. As an illustrative example, the user account may be associatedwith a smart refrigerator. When an object is placed on a shelf in therefrigerator, sensors within the shelf may determine a weight of theobject, and may compare that weight against a previously stored weightfor the object to determine how much of the object was used. Data, suchas sensor data and/or weight data, may be sent to computing system 200from the storage unit/system/device, and may be stored by storagedatabase 296 as inventory data. Computing system 200 may query storagedatabase 296 to determine whether or not that storage device includes aparticular item, and/or an amount remaining of that item. For example,an instructions speechlet may call upon a storage speechlet to accessstorage database 296. Storage database 296 may access inventory dataindicating items stored by one or more storage devices associated with aparticular user account to determine whether or not a particular item isstored, as well as, or alternatively, an amount or quantity of thatparticular item that is being stored. In this way, if instructions datainclude item metadata associated with items needed for a list ofinstructions (e.g., ingredients for a recipe), storage database 296 maybe capable of determining whether the user has that item and/or whetherthere is enough of that item remaining for the task associated with thelist to be completed. For example, if recipe data includes item metadataindicating an item needed to cook a food item, the recipe speechlet maycommunicate with the storage speechlet, which may determine, usingstorage database 296, whether a storage unit/device/system (e.g., asmart refrigerator, smart shelving system) associated with theindividual's user account has that item.

In some embodiments, an application of applications/skills system 262may be written in various computer languages, such as JavaScript andJava. Various applications may include an intent schema file and asample utterances file, such as, for example, IntentSchema.j sonincluding the JSON defining the intents of a particular application, anda SampleUtterances.txt including plain text sample utterances for theparticular application. Applications may also include applicationspecific code, such as a JavaScript or Java file, common library code,and various other JavaScript or Java modules.

TTS system 264 may employ various text-to-speech techniques. However,techniques for transcribing speech into text are well known in the artand need not be described in further detail herein, any suitablecomputer implemented speech to text technique may be used to convert thereceived audio signal(s) into text, such as SOFTSOUND speech processingtechnologies available from the Autonomy Corporation, which isheadquartered in Cambridge, England, United Kingdom. TTS system 264 mayalso include processor(s) 252, storage/memory 254, and communicationscircuitry 256.

Speaker identification system 248, in some embodiments, may correspondto any suitable device/system capable of identifying a particularperson's voice from an audio signal. Speaker identification system 248may determine whether a current voice being used to speak matches knownvoice biometric data associated with a particular individual's voice. Insome embodiments, voice biometric data may be stored within accountssystem 268 for various individuals having a user account stored thereby.For example, individual 2 may have a user account on computing system200 (e.g., stored within accounts system 268), which may be associatedwith electronic device 10. Stored within the user account may be voicebiometric data, such as stored vectors representing stored acousticfeatures associated with a voice of individual 2. Therefore, when anutterance, such as utterance 4, is detected by electronic device 10, andsubsequently when audio data representing that utterance is received bycomputing system 200, speaker identification system 248 may determinewhether the voice used to speak utterance 4 matches, to at least apredefined confidence threshold, the stored voice biometric informationassociated with individual 2 stored by their user account. If so, thenthis may indicate that individual 2 is the likely speaker of utterance4.

In some embodiments, speaker identification system 248 may receive audiodata representing an utterance, or a copy of the audio data, at asubstantially same time as ASR system 258. The audio data may be dividedinto audio frames representing time intervals, with which a number ofvalues or features representing qualities of the audio data may bedetermined, along with a set of those values (e.g., feature vectors oraudio feature vectors) representing features/qualities of the audio datafor each audio frame. For example, each audio frame may include 25 ms ofaudio, and the frames may start at 10 ms intervals. This may result in asliding window where adjacent audio frames include 15 ms of overlappingaudio. Persons of ordinary skill in the art will recognize that manydifferent acoustic features may be determined, and each feature may berepresentative of a particular quality of the audio data. Some exemplaryapproaches that may be used to process the received audio data mayinclude, but art not limited to, mel-frequency cepstral coefficients(“MFCCs”), perceptual linear predictive (“PLP”) techniques, neuralnetwork feature vector techniques, linear discriminant analysis, andsemi-tied covariance matrices. Speaker identification system 248 mayalso include a scoring component that determines respective confidencescores indicating how likely it is that an input utterance was spoken bya particular user.

When audio data is received by computing system 200, ASR system 258,speaker identification system 248, and/or any other suitable componentof speech-processing system 250, may perform windowing functions to theaudio data to generate framed audio data. The size of each audio framemay depend on a particular configuration of speech-processing system250, and persons of ordinary skill in the art will recognize that anyaudio frame size may be used. For example, as mentioned previously, eachaudio frame may include 25 milliseconds of audio data, and may overlapwith 10 milliseconds of a next audio frame, resulting in a slidingwindow. Performing a windowing function may include multiplying a timerecord by a finite-length window with an amplitude that varies smoothlyand gradually toward zero at its edges. By performing windowing,endpoints of the waveforms of respective audio frames of audio datameet, resulting in a continuous waveform without sharp transitions. Afast Fourier transform (“FFT”) may be performed to convert the waveformsin each audio frame of the framed audio data from its original domain(e.g., time) to a representation in a frequency domain (thereby creatingfrequency domain framed audio data). Audio processing techniques otherthan or in addition to FFT may be used to transform audio data (e.g.,waveforms) into data that can be processed as needed.

In some embodiments, user recognition feature extraction may beperformed on the frequency domain framed audio data. User recognitionfeature extraction may include performing frame level feature extractionand/or utterance level feature extraction. The frame level featureextraction may determine which frame of a universal background model(“UBM”) the frame corresponds to. The UBM may be a Gaussian mixturemodel, a deep neural network, etc. The utterance level featureextraction may analyze aligned speech frames to derive feature vectorsof fixed length (i.e., the user recognition feature/vector data). Thefeature extraction may continue until voice activity is no longerdetected in the input audio data, at which point an endpoint of thespeech may be identified and speech processing may end. Featureextraction may, in some embodiments, be performed on all the audio datareceived from the electronic device 10. Alternatively, featureextraction may only be performed on audio data including speech. Featureextraction and user recognition feature extraction may includedetermining values (i.e., features) representing qualities of thefrequency domain framed audio data, along with quantitating thosefeatures into values (i.e., acoustic feature vectors or audio featurevectors). Feature extraction may determine automatic speech recognitionfeature/vector data, which may assist with speech recognition processingfor ASR system 258, and user recognition feature extraction maydetermine user recognition feature/vector data, which may assist withspeaker identification/user recognition for speaker identificationsystem 248. The feature/vector data and the user recognitionfeature/vector data may include the same features/vectors, differentfeatures/vectors, or may include some overlapping features/vectors. Anumber of approaches may be used to extract features/vectors from thefrequency domain framed audio data, such as MFCCs, PLP techniques,neural network feature vector techniques, linear discriminant analysis,semi-tied covariance matrices, and persons of ordinary skill in the artwill recognize that any other suitable approach may be employed.

Speaker identification system 248 may perform speaker identificationusing various data including user recognition features/vector data, andtraining data that may correspond to sample audio data corresponding toknown users associated with a particular device (e.g., electronic device10). Speaker identification system 248 may generate confidence scoresindicating a likelihood that a particular utterance was spoken by one ofthe users associated with a particular device, and may determine whetherany of these confidence scores is greater than a predefined confidencescore threshold. If so, then that may indicate a likelihood that acertain user is the speaker of the utterance. If two or more confidencescores are determined to be in excess of the confidence score threshold,then speaker identification system 248 may select the user having thegreater confidence score, or may prompt the device to obtain additionalinformation to resolve the speaker's identity.

In some embodiment, training data may be obtained and stored by accountssystem 268. The training data may be stored as waveforms and/orcorresponding features/vectors, and may correspond to data from variousaudio samples, each audio sample associated with a known user and/oruser identity. For example, each user known to the system may beassociated with some set of training data for the known user. Speakeridentification system 248 may then use the training data to compareagainst incoming audio data (represented by user recognitionfeature/vector data) to determine an identity of a user speaking anutterance. The training data may be associated with multiple users ofmultiple devices and therefore may be associated with both a user thatspoke the respective utterance, as well as electronic device 10, whichprovided the audio data representing the spoken utterance.

The training data for a particular user may include a feature vector ofthe same size as a vector of the user recognition feature/vector data.Thus, for example, if a feature vector is of size F, the training datamay also be a feature vector of size F. To create such a training datafeature vector, during a training period computing system 200 may eitherprompt a user to speak sample audio data or may identify sample audiodata known to have been spoken by a particular user. The system may thenprocess the sample audio data to create sample training data (e.g., afeature vector of size F). The training data may then be stored byaccounts system 268 and saved for use during runtime user verificationprocessing.

In some embodiments, speaker identification processing may furtherinclude various additional techniques to determine a speaker of anutterance. For example, device beaconing may indicate a likelihood thata particular individual's device is located proximate to electronicdevice 10. In this particular scenario, if an individual's device isdetermined to be substantially close to electronic device 10 at a timewhen the wakeword is uttered, then this may indicate that the wakewordwas likely spoken by that individual. As another example, historicalrules may be employed to determine a speaker of an utterance. Forexample, one particular individual may typically interact withelectronic device 10, and therefore it may be more likely that when awakeword is detected by electronic device 10, that wakeword is morelikely than not spoken by that particular individual. Still further,certain individuals may be interact with electronic device 10 duringcertain times, or to perform certain functions. For example, aparticular individual may typically ask for a weather forecast usingelectronic device 10 during a particular time frame (e.g., between 8 and9 o'clock). In this particular scenario, if the wakeword is determinedto have been uttered during that time frame, then this may indicate thatthere is a high likelihood that a speaker of the wakeword (and thecorresponding utterance) is that individual. As another example, if aparticular individual has just performed a specific function, such assending a message to another individual, then if the wakeword isdetected by electronic device 10 temporally after, and within a certainamount of time of, the specific function, then this may indicate thatthat individual likely spoke the corresponding utterance. Persons ofordinary skill in the art will recognize that various additionaltechniques may be employed to determine a speaker of an utterance, andthe aforementioned are merely exemplary. Speaker identification system248 may also include processor(s) 252, storage/memory 254, andcommunications circuitry 256.

Accounts system 268 may store one or more user accounts or userprofiles, corresponding to users having an account on computing system200. For example, a parent may have an account registered on computingsystem 200, and each of the parent's children may have their own userprofile registered under the parent's account. In some embodiments, eachaccount may be identified by a particular account identifier. When arequest (e.g., audio data representing an utterance, text data, etc.) isreceived by computing system 200, an account identifier associated withthat request may also be received, which may specify an accountidentifier associated with a device that the request was received from.Information, settings, and/or preferences, for example, for each userprofile may be stored by accounts system 268.

In some embodiments, accounts system 268 may store a voice signal, suchas voice biometric information, for a specific user profile. This mayallow speaker identification techniques to be used to match a voice tovoice biometric data associated with a specific user profile. In someembodiments, accounts system 268 may store a telephone number assignedto a particular user profile. In some embodiments, accounts system 268may include a list of media items currently stored within anindividual's registered account or user profile. For example, a list ofmusic or videos purchased or obtained by an individual may be storedwithin the individual's user profile on accounts system 268, which maybe accessed by the individual when the individual seeks to hear aparticular song or songs, or view a particular video or videos. Accountssystem 268 may also include a listing of all applications currentlyenabled for each user profile. In some embodiments, NLU system 260 mayreceive indications of which applications are currently enabled for aparticular user profile or account, such that NLU system 260 is aware ofwhich rules and capabilities that computing system 200 is able toperform for the particular user profile or account.

In some embodiments, accounts system 268 may include a device database290. Device database 290 may store device information associated witheach user account. For example, device database 290 may store deviceidentifiers (e.g., IP addresses, MAC addresses, serial numbers, etc.)for various devices associated with a particular user account. Forexample, devices 286A-N may be associated with a first user account,devices 288A-N may be associated with a second user account, and devices294A-N may be associated with an n-th user account.

Orchestrator 250, in a non-limiting embodiment, may correspond to anysuitable device, system, and/or logic capable of performing one or moreprocesses. For example, orchestrator 250 may be configured to requestcontextual metadata from one or more domains/applications ofapplications/skills system 262 in response to receive a notificationfrom electronic device 10 that content is currently being renderedthereby. In response to receiving the contextual metadata, orchestrator250 may be configured to provide the contextual metadata to NLU system260. In one embodiment, orchestrator 250 may include processor(s) 252,storage/memory 254, and communications circuitry 256.

Persons of ordinary skill in the art will recognize that although eachof speaker identification system 248, ASR system 258, NLU system 260,applications/skills system 262, TTS system 264, accounts system 268, andorchestrator 250 may each include instances of processor(s) 252,storage/memory 254, and communications circuitry 256, and thoseinstances of processor(s) 252, storage/memory 254, and communicationscircuitry 256 within each of speaker identification system 248, ASRsystem 258, NLU system 260, applications/skills system 262, TTS system264, accounts system 268, and orchestrator 250 may differ. For example,the structure, functionality, and style of processor(s) 252 within ASRsystem 258 may be substantially similar to the structure, functionality,and style of processor(s) 252 within NLU system 260, however the actualprocessor(s) 252 need not be the same entity.

FIG. 3A is an illustrative diagram of an exemplary system for causingsupplemental content to be rendered by a device based on content that isbeing rendered by the device, in accordance with various embodiments. Inthe illustrative embodiment, a task may be performed using embeddedinstructions for a voice user interface (“VUI”). For example, individual2 may be cooking a food item, or more generally performing a task, usingan interactive recipe including embedded instructions, where a portionof the recipe being displayed by display screen 212 of electronic device10.

In the illustrative embodiment, electronic device 10 may render a GUI312 may be displayed by display screen 212, which may function inconjunction with a VUI of electronic device 10. The VUI may allowelectronic device 10 to function primarily through speech interactions,however persons of ordinary skill in the art will recognize that thismerely exemplary. In particular, GUI 312 may display a step of therecipe where a particular action is to be performed. For example,individual 2 may be cooking a recipe that includes a step of separatingeggs. If a user does not know, the user may inquire as to how to performa particular step. For example, the recipe data may include step datarepresenting a step of the recipe indicating that eggs are to beseparated. Furthermore, display data representing GUI 312 may begenerated and displayed by electronic device 10 based on step datarepresenting a step of the recipe, where the step data includes textdata indicating the action (e.g., “separate eggs” step) to be performedby individual 2. If individual 2 does not know how to perform the actionassociated with that step, individual 2 may say utterance 304, “Alexa,how do you separate eggs?” In response to determining that the wakeword(e.g., “Alexa”) was uttered, electronic device 10 may package and sendaudio data representing utterance 304 to computing system 200. Computingsystem 200, upon receipt of the audio data, may generate text datarepresenting the audio data using ASR system 258.

Orchestrator 250, in one embodiment, may provide the text data to therecipe speechlet, which may compare the text data to expected utterancetext data representing an expected utterance associated with aparticular step. When the recipe data is being generated, a recipebuilder may include, for certain steps, metadata indicated one or moreutterances that an individual may say for that step. For example, for astep corresponding to “separating eggs,” the recipe builder may includean expected utterance of “How do you separate eggs?”, or “Can you showme how to separate eggs?” Therefore, if an utterance is spoken while aparticular step of a recipe is being rendered, then the recipe speechletmay be configured to compare the utterance to the expected utterance todetermine whether or not they are similar.

Alternatively, or additionally, in some embodiments, orchestratorcomponent 250 may provide the text data to NLU system 260. Notificationdata may be provided from electronic device 10 to computing system 200that indicates that content is currently being rendered by displayscreen 212. In response to receiving the notification, orchestrator 250may request that a domain responsible for providing the content identifyitself, and send entity data representing the content to NLU system 260.As an illustrative example, a recipe domain may be responsible for therendered content (e.g., GUI 312), and therefore may send entity datarepresenting content including GUI 312 to NLU system 260. In anotherembodiment, orchestrator 250 may track which domain is currentlyresponsible for providing content to electronic device 10, and may, uponreceiving the notification data, obtain entity data representing therendered content from that domain. Using the entity data and the textdata, NLU system 260 may determine that the intent of utterance 304 isfor assistance with a particular step of a list of instructions beingrendered by electronic device 10, and may pass output data includingslots associated with the intent and the entity data to a recipespeechlet associated with the recipe domain.

The recipe speechlet may determine that the intent of utterance was arequest for assistance in performing a step of the recipe. The recipespeechlet may determine the particular step that is being requested tobe performed based on the content it is providing as well as its ownmonitoring of the current state of the recipe. For example, the recipespeechlet may determine that a current step of the recipe is step5—“Separate Eggs”—based on the fact that the recipe speechlet hadrecently provided display data representing GUI 312 based on step datarepresenting step 5 to electronic device 10. As an illustrative example,the recipe domain 222 may be configured to facilitate actions associatedwith a recipe, or generally a list of instructions, using a recipespeechlet. In the illustrative embodiment, first recipe data 320representing a first recipe—“Recipe 1”—may the current list ofinstructions being rendered by electronic device 10. Recipe data 320 mayinclude first step data 322 corresponding to step 4 of recipe 1—“BoilWater”—and second step data 324 corresponding to step 5 of recipe1—“Separate Eggs.”

In some embodiments, recipe data 320 may also include recipe metadata330. Recipe metadata 330 may include supplemental information associatedwith one or more steps, or aspects, of recipe data 320. For instance,step data 324 may include an expected utterance “How do you separateeggs?” When the text data representing utterance 304 is received by therecipe speechlet, a similarity score may be generated between the textdata representing utterance 304, and text data representing the expectedutterance. If the similarity score is greater than a similarity scorethreshold value, then that may indicate that utterance 304 correspondsto one of the expected utterances associated with step data 324.

In response to determine the intent of utterance 304, whether it beusing NLU processing or by determining that the similarity score exceedsthe similarity score threshold, step metadata associated with step data324 may be accessed. For instance, first link metadata 332 and/or secondlink metadata 334 may be accessed, which may correspond to URL links toa first video 326 and a second video 328, respectively. Each of firstvideo 326 and second video 328 may be provided by the recipe builderwhen generate the recipe data to assist the individual in performing anaction associated with second step data 324. As an illustrative example,if step 5 of recipe 1 is for eggs to be separated, first video 326entitled “How to separate eggs,” and second video 328 entitled“Separating eggs,” may both be available for that step, as second stepdata 324 includes first and second link metadata 332 and 334, whichindicate links to the aforementioned videos.

In some embodiments, the metadata associated with a particular step maybe curated by a builder of that list of instructions. For example, anindividual that created recipe data 320 may include first link metadata332 and second link metadata 334 with second step 324. In this way, whena request is received for assistance associated with second step data324, the recipe speechlet is able to select one of the provided metadatalinks. Further still, if two or more pieces of metadata are provided,then upon a request being received that is to use the metadata, therecipe speechlet may be configured to select between one of the two ormore pieces of metadata. The recipe speechlet, for example, may beconfigured to select metadata based on number of times a particularpiece of metadata is used, a preference of a user, and/or a preferenceof a recipe builder.

In the illustrative embodiment, computing system 200 may be configured,after identifying the intent of utterance 304 and obtaining theappropriate metadata for the intent, to generate and send display datarepresenting a new GUI 314. For example, in response to determining thatutterance 304 correspond to a request for how to separate eggs, a URLassociated with first video 326 may be accessed, and the URL may beprovided to electronic device 10 such that video 326 may be streamedfrom video domain 224 to electronic device 10. In some embodiments, therecipe speechlet may further be configured to generate display datarepresenting GUI 314, which may include first video 326.

In some embodiments, computing system 200, and in particular the recipespeechlet, may be configured to determine text data representing amessage that is to be output after identifying content to provide inresponse to determine the intent of utterance 304. For example, therecipe speechlet may include text data representing a message indicatingthe video to be output and the subject that it is related to. The recipespeechlet may be configured to provide the text data to TTS system 264,which may generate audio data representing the text data, and the audiodata may be sent to electronic device 10. For instance, response306—“Here is a video showing how to separate eggs”—may be output inresponse to first video 326 beginning to be streamed by electronicdevice 10.

FIG. 3B is an illustrative diagram of an exemplary system fordetermining that an item associated with a list of instructions isabsent, and causing that item to be ordered, in accordance with variousembodiments. In the illustrative embodiment, individual 2 may speakutterance 354, where individual 2 and electronic device 10 may belocated within a same environment. Utterance 354, in one embodiment, maybe “<Wakeword>, <Perform Instruction Intent> <Instruction Title>,” where<Wakeword> corresponds to an exemplary wakeword for electronic device 10(e.g., “Alexa”), <Perform Instruction Intent> corresponds to anexemplary task to be performed (e.g., cook a pizza), and <InstructionTitle> corresponds to an exemplary title of the task. For example, thetask may be to make a pizza. In this particular scenario, theinstruction intent may be “make” and the instruction title may be“pizza.”

In response to determining that the wakeword has been uttered,electronic device 10 may begin packing and sending audio datarepresenting utterance 354 to computing system 200. Upon determining anintent, which may be for a task to be performed, of utterance 354,computing system 200 may identify a list of instructions 360corresponding to the task. For example, if the task is to make pizza,then the list of instructions may correspond to a recipe to make pizza.

As described previously, instructions database 280 may include list datarepresenting a list of instructions, such as, for example, a recipe.Instructions database 280 may also store metadata associated with thelist data. For instance, list data 370, which may be stored byinstructions database 280, may represent a list of instructionsincluding one or more instructions to be performed to complete a task(e.g., steps to cook a food item). List data 370 may include instructiondata representing various instructions of the list, and list data 370may also include metadata associated with particular instruction data(e.g., metadata associated with first instruction data representing afirst instruction of the list, second instruction data representing asecond instruction of the list, etc.). For example, first instructiondata representing a first instruction of the list may include firstinstruction metadata indicating one or more URL links detailing how toperform a particular instruction.

In some embodiments, list data 370 may also include metadata that isunassociated with any one particular instruction. For instance, thismetadata may be associated with a preamble portion of list data 370. Forexample, list data 370 may include item metadata, which may indicate oneor more items 362 needed for the task associated with list data 370. Theitem metadata may be provided, in one embodiment, by a builder whengenerating the list data. As an illustrative example, the list data maycorrespond to recipe data representing a recipe, and the item metadatamay indicate ingredients needed to cook the food item associated withthe recipe.

In some embodiments, computing system 200 may determine that the useraccount associated with electronic device 10 is also associated with oneor more storage devices/system. For example, the user account may beassociated with one or more smart storage devices/systems, such as, butnot limited to, a smart refrigerator or a smart shelving unit. Each ofthese storage units may be capable of monitoring and tracking, amongstother features, a type of item stored within that storage unit, anamount (e.g., percentage, weight, etc.) remaining of a particular item,and a location of an item within the storage unit. For example, IoTdevice 20 may correspond to a smart refrigerator, which may be capableof tracking items stored by the smart refrigerator.

In some embodiments, a storage database may store inventory datacorresponding to item information associated with the contents of theone or more storage units/devices/systems associated with the useraccount. For instance, storage database 296 may store first inventorydata representing a first storage unit status 364 associated with afirst storage unit. Therefore, information associated with the contentsof the first storage unit (e.g., IoT device 20) may be accessible bycomputing system 200 using storage database 296, and may the inventorydata may indicate which items are, or have been, stored by a firststorage unit (e.g., a smart refrigerator), as well as an amount of itemcurrently stored within the first storage unit. For example, firststorage unit status 364 may indicate that a first storage unit isstoring Item 2, Item 4, and Item 5, and the amounts remaining of each ofthese items may be 100% (e.g., full), 50% (e.g., half full), and 0%(e.g., empty). Similarly, a second storage unit status, corresponding toa second storage unit (e.g., a smart refrigerator, smart shelving unit,etc.) may also be accessible by computing system 200, and may indicatewhich items are, or have been, stored by the second storage unit, aswell as an amount of each item that is currently stored thereby. Forexample, storage database 296 may store second inventory datarepresenting a second storage unit status 366, which may indicate thatthe second storage unit may be store Item 1, Item 3, and Item 6, andthat the amount remaining of these items are 0% (e.g., empty), 90%(e.g., 90% full), and 50% (e.g., half full), respectively.

An instructions speechlet responsible for executing the actionsassociated with the particular list of instructions being rendered mayreceive list data 370 representing a list of instructions, and maydetermine that list data 370 includes item metadata 372 corresponding toitems needed for the task associated with the list of instructions. Theinstructions speechlet may be configured to determine whether theseitems are available (e.g., that there is enough of the item to be usedfor the list) for use in performing the task using the item metadata.For example, upon receiving list data 370, the instructions speechletmay determine that list data 370 includes item metadata 372, and may beconfigured to access a user account associated with the requestingdevice to determine whether or not those items are currently stored byany storage units/devices/systems associated with that user account.

The instructions speechlet, in some embodiments, may pass item datarepresenting items to be searched for within a storage unit to a storagespeechlet. The storage speechlet may query storage database 296 todetermine which of the inquired items are stored by the storage unitsassociated with the user account. In some embodiments, the storagespeechlet may further search for an amount of an item. As anillustrative example, item metadata 372 may indicate that Item 1, Item2, and Item 3 are needed for the task associated with list data 370. Thestorage speechlet may query storage database 296 to determine whether afirst storage unit and a second storage unit currently is storing theseitems. In the illustrative example, the storage speechlet may determinethat Item 2 and Item 3 are available, but Item 1 is unavailable. Thisinformation may be passed back to the instructions speechlet, which maybe configured to perform and/or execute one or more actions in response.

In some embodiments, if one or more items are determined to beunavailable, computing system 200 may be configured to ask therequesting individual (e.g., individual 2) whether or not he/she wouldlike to purchase that item. For example, the instructions speechlet maydetermine text data representing a message indicating that an item(e.g., “Item 1”) is needed for the task (e.g., “Instructions Title”),and inquiring as to whether individual 2 would like that item to beordered. For instance, text data representing a message—“For<Instructions Title> you need <Item 1>. Do you want me to order it?”After determining the text data, the instructions speechlet may provideor otherwise cause the text data to be sent to TTS system 264 togenerate audio data representing the text data. The audio data may thenbe sent from computing system 200 to electronic device 10, and output asoutput audio message 356.

In response to output audio message 356, individual 2 may respond withan utterance 358, “Yes.” This may indicate that individual 2 would liketo have the item ordered. In some embodiments, individual 2 may speakthe wakeword or other activation mechanism prior to speaking utterance358 (e.g., “Alexa, yes”), however persons of ordinary skill in the artwill recognize that this is merely exemplary. Upon receipt of audio datarepresenting utterance 358, text data representing the audio data may begenerated and provided to the instructions speechlet and/or NLU system260. The intent of utterance 358 may be determined to be an affirmativeresponse to the order inquiry of output audio message 356, and this maycause the instructions speechlet to determine text data representinganother message asking for a time that the item should be delivered.This text data may be provided to TTS system 264, thereby generatingaudio data, which may be sent to electronic device 10 and output asoutput audio message 360. In response to output audio message 360,individual 2 may speak an utterance 362 indicating a time that he/shedesires the order to be delivered, “7:00.” In response to receivingaudio data representing utterance 362, text data representing this audiodata may be generated and provided to the instructions speechlet and/orNLU system 260. The intent of utterance 362 may be determined toindicate the time that the order is to be delivered, and this may causethe instructions speechlet to determine text data representing yetanother message indicating to individual 2 that the order is beingplaced and confirm the delivery time. For instance, audio datarepresenting the text data may be generated, and the audio data may besent to electronic device 10 and output as output audio message 364,“Ok. Your order will be here by 7:00.”

Upon determining that the item is to be ordered, the instructionsspeechlet may access an item purchasing system, which may also bereferred to as a purchasing fulfillment system/service, to purchase thatitem as well as have that item delivered. In some embodiments, theinstructions speechlet may generate purchasing data representing thepurchasing intent to buy the indicated item (e.g., “Item 1”), and maysend the purchasing data to the fulfillment system/service. Furthermore,in some embodiments, upon receiving list data 370 and determining thatlist data 370 includes item metadata 372, the instructions speechlet mayask individual 2 whether he/she would like to purchase all of the itemsneeded for the task associated with the list of instructions. If so,then the instructions speechlet may generate purchasing data to procurethe items indicated as being needed for list data 370 by item metadata372. For instance, the list data may include item metadata indicating afirst plurality of items needed for the task (e.g., a recipe). The itemmetadata may also indicate amounts of those items needed (e.g., onecarrot, two eggs, etc.). The item metadata may then be used to purchaseall of the items needed for the task such that the entire set of itemsneeded for the task may be delivered to individual 2.

In some embodiments, the instructions speechlet may be configured todetermine display data to be provided to electronic device 10 indicatingthe items needed for list data 360. For instance, in response todetermining that list data 360 representing the list includes itemmetadata 372, the instructions speechlet may determine and/or generatedisplay data representing GUI 316, and computing system 200 may send thedisplay data to electronic device 10 to be rendered by display screen212. GUI 316 may include some or all of the items indicated by metadata362 such that the items may be displayed on display screen 212 such thatindividual 2 may be capable of viewing the items.

FIGS. 4A and 4B are illustrative flowcharts of an exemplary process fordetermining an action to be performed by a first device based on listdata and audio data output by a second device, in accordance withvarious embodiments. Process 400, in a non-limiting embodiment, maybegin at step 402. At step 402, audio data representing a firstutterance may be received from a first device. For example, computingsystem 200 may receive audio data representing utterance 4 fromelectronic device 10. In some embodiments, audio data representing anutterance may be sent to computing system 200 in response to a wakeword,trigger expression, or sound, being detected by electronic device 10.For example, in response to individual 2 saying the wakeword “Alexa,”electronic device 10 may begin sending audio data representing soundscaptured by one or more microphones of electronic device 10 to computingsystem 200. However, in some embodiments, a manual input being detectedby electronic device 10 may alternatively cause electronic device 10 tobegin capturing and sending audio data representing sounds received byone or more microphones of electronic device 10 after the manual inputis detected.

At step 404, an account identifier may be received. For instance, anaccount identifier may be received by computing system 200 fromelectronic device 10 in response to audio data representing an utterancebeing sent to computing system 200. In some embodiments, the accountidentifier may be sent to computing system 200 with the audio data, suchas the audio data mentioned previously for step 402. For example, theaccount identifier may be sent as metadata appended to the audio data.At step 406, a user account associated with the account identifier maybe determined. For example, accounts system 268 may be queried with theaccount identifier, and a user account associated with that accountidentifier may be determined.

At step 408, first text data representing the first audio data may begenerated. Upon receipt of the audio data, orchestrator 250 of computingsystem 200 may provide the audio data to ASR system 258. ASR system 258may, using grammar rules and models associated therewith, generate textdata representing the audio data. Orchestrator 250 may then provide thetext data to NLU system 260. At step 410, NLU system 260 may determinethat the utterance corresponds to a first intent. For example, NLUsystem 260 may determine that the utterance corresponds to a request tofor a recipe to make pizza, or instructions to build a birdhouse, andthe like. In some embodiments, a notification may further be received bycomputing system 200 from electronic device 10 that indicates thatcontent is currently being rendered by electronic device 10. In thisparticular scenario, orchestrator 250 may request that a domainresponsible for providing the content being rendered by electronicdevice 10 identify itself, and provide entity data representing thecontent to NLU system 260. After the domain identifies itself and sendsthe entity data, NLU system 260 may rank the various intent hypothesisfor the utterance using the entity data and the text data to determinethe first intent.

At step 412, declared slots associated with the first intent may bedetermined. For instance, if the intent corresponds to a recipe intent,some exemplary declared slots may be a “Food Item To Be Made” slot, a“Recipe Title” slot, a “Chef” slot, and/or a “Recipe Builder” slot. Asan illustrative example, text data representing the utterance, “I wantto make ‘Pizza’ by ‘Chef 1’,” may be parsed and tagged by NLU system260. The result may be attributed the value “Pizza” to the “Food Item ToBe Made” slot, and the value “Chef 1” to the “Chef” slot. However,persons of ordinary skill in the art will recognize that theaforementioned is merely exemplary. At step 414, available slots may bedetermined using the first text data. For instance, using the previousexample, the available slots may be the “Food Item To Be Made” slot andthe “Chef” slot, however different slots may be applicable for differentintents, and different available slots may be determined for differentutterances.

At step 416, instructions database 280 may be queried. For instance,using the values associated with the available slots determined at step414, instructions database 280 may be searched for lists of instructions(e.g., one or more of lists 282A-N) that correspond to the intendedinstructions requested by the utterance. For example, if the utteranceis, “I want to make pizza using ‘Chef's’ recipe,” instructions database280 may search through lists 282A-N for lists that include an item to bemade being pizza (e.g., {Item To Be Made}: “Pizza”) and a chef being“Chef” (e.g., {Chef}: “Chef”).

At step 418, list data representing a first list of instructions may bereceived. The first list of instructions may correspond to instructionsstored by instructions database 280 that corresponds to the instructionsdesired by the utterance. For example, the list of instructions maycorrespond to a recipe (e.g., a pizza recipe), and therefore recipe datarepresenting the recipe may be received. In some embodiments, if two ormore lists of instructions are determined to match the intended result,then computing system 200 may prompt a requesting device to selectbetween the two or more lists. For example, if individual 2 wants tomake pizza, and instructions database 280 includes a first pizza recipeand a second pizza recipe, computing system 200 may generate a messageto be sent to electronic device 10 to selected between the two pizzarecipes. The message may be audio and/or video, and may indicate toindividual 2 that two recipes were found substantially matching therequested recipe. In this particular scenario, individual 2 may selectone of the two prompted recipes via voice and/or manual input (e.g.,touching a touch screen), and computing system 200 may be configured torender recipe data representing the selected recipe in response.

At step 420, display data representing content including the first listof instructions may be generated. For example, list data representingthe list of instructions may include, or may indicate, a GUI to bedisplayed by a requesting device. For instance, the list data mayinclude instructions data representing an instruction of the list, andthe instructions data may include text data representing an action to beperformed by a human. Upon identifying the list data, the instructionsspeechlet may be configured to generate display data that represents atleast a portion of the list (e.g., one or more instructions). At step422, the display data may be sent to the first device. For example,computing system 200 may send display data representing GUI 112 toelectronic device 10. Upon receipt, electronic device 10 may display GUI112 using display screen 212. The list data may include firstinstruction data corresponding to a first instruction of the list. Forexample, recipe data may include first step data representing a firststep of the recipe. The first instruction data may include text datathat is intended for presentation to a human and describes a firstaction for the human to do to prepare the first food item, and the textdata may be included within the display data provided to electronicdevice 10.

At step 424, second audio data representing a second utterance may bereceived from the first device. For example, second audio datarepresenting utterance 4 may be received by computing system 200 fromelectronic device 10. As mentioned previously, in some embodiments, thesecond audio data may be sent to computing system 200 in response toelectronic device 10 determining that a wakeword (e.g., “Alexa”) hadbeen uttered within a local environment where electronic device 10 islocated. At step 426, notification data indicating that content is beingrendered by the first device may be received by computing system 200. Insome embodiments, in response to an utterance including a wakeword, atrigger expression being detected, and/or a manual input being detected,electronic device 10 may be configured to determine a current devicestate. The current device state may correspond to a status of whether ornot content is currently being provided to electronic device 10, and/orwhether or not content is currently being rendered by electronic device10. Content, as described herein, may refer to display content (e.g., aGUI, a web-view, an image, etc.) capable of being displayed by a displayscreen, and/or audio content (e.g., an audio file) capable of beingoutput by an audio output component. Upon the trigger (e.g., wakeword,trigger expression, manual activation) being detected, electronic device10 may determine whether or not any content is being rendered thereby,and send notification data to computing system 200 indicating whether ornot content is being rendered. For example, a TRUE/False, or logical 1/0may be sent to computing system 200, where TRUE or a logical 1 mayindicate that content is being rendered, whereas FALSE or a logical 0may indicate that content is currently not being rendered. However,persons of ordinary skill in the art will recognize that any suitablemetadata reflecting the current device state may be employed, and theaforementioned is merely exemplary. At step 428, second text datarepresenting the second audio data may be generated. For instance, uponreceipt, orchestrator 250 may provide the audio data to ASR system 258,which in turn may generate text data representing the audio data.

At step 430, computing system 200 may determine that the secondutterance is associated with a first domain. For instance, orchestrator250 may determine that the instructions domain is currently responsiblefor providing content to electronic device, and may determine that theintent of the second utterance is likely associated with theinstructions domain. Alternatively, the text data may be provided to NLUsystem 260 which may determine that an intent of the utterance isassociated with the instructions domain.

At step 432, entity data representing the rendered content may bereceived. For instance, upon receiving the notification data,orchestrator 250 may send a request to the various domains associatedwith computing system 200. The request may ask that any domainresponsible for providing content to electronic device 10 identifyitself, as well as provide entity data representing that content to NLUsystem 260. In some embodiments, each domain may be configured to formatcontent to be provided to devices in a manner consistent with thevarious slots associated with intents related to that domain. Forexample, the music domain may include intents associated with listeningto music, and these intents may include such slots as “Song Title”slots, “Album Title” slots, and “Artist Title” slots. Therefore, contentrendered by a functionality associated with the music domain may beformatted to, if available, include “Song Title” slots, “Album Title”slots, and “Artist Title” slots and their associated values. However,persons of ordinary skill in the art will recognize that theaforementioned is merely exemplary and, alternatively, orchestrator 250may request the entity data from the first domain based on orchestrator250 knowing that the first domain is currently responsible for providingthe rendered content to the first device. As an illustrative example, ifa recipe domain is currently responsible for providing content toelectronic device 10 (e.g., display data representing a recipe), thenthe recipe domain may send entity data representing that content (e.g.,entity data representing GUI 112) to NLU system 260. In someembodiments, the first domain may provide the entity data toorchestrator 250, which may in turn provide the entity data to NLUsystem 260.

At step 434, contextual slots may be extracted from the first entitydata. For instance, contextual resolution system 270 may be configuredto receive the entity data, and determine what slots are included withinthat entity data. For example, if the entity data representing GUI 112is received by contextual resolution system 270, the entity data mayindicate that a “Recipe Name” slot, a “Step” slot, and a “Step Action”slot are included. The “Recipe Name” slot, for instance, may correspondto the recipe name of a recipe being displayed by GUI 112 (e.g.,“Pizza”), the “Step” slot may correspond to a particular step of therecipe that is currently being, or to be, performed, (e.g., “Step 1”),and the “Step Action” slot may correspond to an action to be performedby the current step (e.g., “Preheat the oven to 400 F”).

At step 436, the instructions speechlet may determine that the list dataincludes first instruction data representing a first instruction of thelist of instructions. For example, if the list data corresponds torecipe data representing a recipe, then a recipe speechlet may determinethat the recipe data include first step data representing a first stepof the recipe. In some embodiments, the first instruction data mayinclude text data that is intended for presentation to individual 2 anddescribes a first action for individual to do to complete the task. Forexample, the first instruction data may include text data representing amessage “Pre-Heat Oven To 400 Degrees.”

At step 438, the second utterance may be determined to correspond to thefirst instruction data. For example, upon receipt of the second textdata, the instructions speechlet may determine expected utterancesassociated with the first instruction data. The second text data may becompared to text data representing the expected utterance(s), and asimilarity score between the two may be generated. If the similarityscore is greater than a predefined similarity score threshold, then thismay indicate that the second utterance corresponds to one of theexpected utterances. Alternatively or additionally, NLU system 260 mayfurther determine that the second intent corresponds to a first step ofthe first list of instructions being rendered by electronic device 10using the contextual slots extracted from the first entity data. Forexample, if the instruction is to “pre-heat the oven” then the utterancemay be determined to correspond to a “Cause Oven To Perform Action”intent. As another example, if the step is “separate eggs,” then theutterance may correspond to a “Request For Assistance” intent.

At step 440, first instruction metadata associated with the firstinstruction data may be determined. Various instructions of a list ofinstructions may include associated metadata. For example, if a firststep of a recipe is for an oven to be heated, then metadata associatedwith that step may indicate a temperature that the oven is to be heatedto. As another example, if the step is to separate eggs, the metadatamay indicate various videos, or URL links to videos, that can show auser how to accurately perform that action. At step 442, theinstructions speechlet may determine that the first instruction datacorresponds to an action to be performed. In one embodiment, based onthe slots and values associated with those slots from intent resolutionperformed by NLU system 260, the first step may be determined tocorrespond to an action. For example, the first step may be to pre-heatan oven, turn on a machine, or cause another device/system to perform aspecific action. As another example, the first instruction datarepresenting the first instruction may be associated with firstinstruction metadata indicating an action to be performed, a device typeassociated with the action, and/or items needed for that action.

At step 444, a determination may be made as to whether the user accountincludes a device associated with the action to be performed. Forexample, the first instruction metadata may indicate a type of deviceassociated with the first action. Using the first instruction metadata,the user account may be accessed to determine whether any additionaldevices are associated with the user account of the same type of device.If, at step 444, it is determined that there is a second deviceassociated with the action to be performed that is associated with theuser account, then process 400 may proceed to step 446. For instance,computing system 200 may determine that a second device associated withthe user account is capable of performing the action. In someembodiments, if the action to be performed involves a specific device,at step 442, then the instruction data may be associated with metadatathat indicates a device type capable of performing that action.Continuing the aforementioned example, the instructions speechlet maydetermine that a smart oven is needed to execute the action associatedwith a first instruction. The instructions speechlet may also determinethat the user account with which the first device is associated may alsobe associated with a second device capable of performing that action.For example, if IoT device 20 corresponds to a smart oven, then smartoven 20 may be capable of being used to perform the action associatedwith a first step of a recipe.

At step 446, command data may be generate that includes, at least, thefirst instruction metadata. For example, the recipe speechlet maygenerate command data representing a “pre-heat oven” command thatincludes first metadata indicating the first temperature to be used forthe pre-heat oven action. At step 448, third text data representing afirst message indicating that the action is being performed by thesecond device. For example, if the action is for an oven to bepre-heated, and the user account includes a smart oven, then the messagemay be “Ok. Pre-heating your oven.” For instance, the instructionsspeechlet may determine that, in response to the command data beingsent, that the first device is to be updated to indicate that the actionassociated with the first instruction data is being performed. At step450, third audio data representing the third text data may be generated.For example, the third text data may be provided to TTS system 264,which may generate the third audio data.

At step 452, the command data may be sent to the second device. Forinstance, the instructions speechlet may send the command data to IoTdevice 20. Furthermore, at step 454, the third audio data may be sent tothe first device. For example, the message indicating that the action isbeing performed by the second device may be sent to electronic device10. For instance, audio data representing message 6, “Ok. Here is thepizza recipe. Pre-heating your oven to 400 F,” may be sent to electronicdevice 10 from computing system 200. In some embodiments, steps 452 and454 may occur substantially in parallel, however this is merelyexemplary.

If, at step 444, it is determined that there are no devices associatedwith the user account that are also associated with the action (e.g., nodevices of a type of device associated with the action), then process400 may proceed to step 456. At step 456, fourth text data representinga message indicating the action to be performed by the individual may bedetermined. For example, this message may be, “Please set your oven to400 degrees.” As there are no IoT devices associated with the useraccount, individual 2 may need to perform the action. At step 458,fourth audio data representing the third text data may be generated. Forinstance, the fourth text data may be provided to TTS system 264, whichmay generate fourth audio data. At step 460, the fourth audio data maybe sent to the first device. For example, electronic device 10 mayoutput the message in response to receiving the fourth audio data.

FIG. 5 is an illustrative diagram of an exemplary graphical userinterface including an instructions builder for generating instructiondata, in accordance with various embodiments. Instructions builderinterface 500, in a non-limiting example embodiment, may include a textportion 502 capable of being written and edited by an individual. Forexample, computing system 200 may include software allowing individualsto develop and publish instruction data using instruction builderinterface 500. In some embodiments, text portion 502 may be formattedusing an XML language, however any suitable programming language/systemmay be employed. In the illustrative non-limiting embodiment, the listof instructions may correspond to a recipe. In this particular scenario,the list data may correspond to recipe data, and the instruction datarepresenting instructions of the list of instructions may correspond tostep data representing steps of the recipe. However, persons of ordinaryskill in the art will recognize that this is merely exemplary.

In the illustrative embodiment, text portion 502 may include a firsttext portion 504 and a second text portion 506. First text portion 504may include information associated with a first step of a list ofinstructions, and second text portion 506 may include informationassociated with a second step of the list. For example, first textportion 504 may correspond to first step data while second text portion506 may correspond to second step data. Each text portion may indicatevarious features associated with that step. For example, the first stepof the list of instructions is “Step 1—“Pre-heat oven to 400 degreesFahrenheit.” In this particular example, various entities may beattributed values by a recipe builder such that an action associatedwith the first step may be performed. In some embodiments, first textportion 504 may also include metadata associated with the first step.For example, the metadata may indicate that a device type correspondingto an “oven” may be needed for the first step.

In some embodiments, a menu of available device entities 508 may beprovided to the recipe builder. The recipe builder may be capable ofdragging and dropping a particular entity into slot data for aparticular step such that the corresponding entity is identifiable byNLU system 264. In one embodiments, instruction builder interface 500may be capable of recognizing entities within text as the text is beinginput. For example, as an individual starts typing the letters “o-v,”interface 500 may recognize that the individual may be typing out theentity “oven,” and may allow the individual to use the entity withoutthe need to complete the remaining letters. For example, the individualmay accept the suggested entity “oven” for use within first text portion504.

Second text portion 506 may, in the illustrative embodiment, correspondto second step data representing a second step of the list ofinstructions. Second text portion 506 may include entities attributedwith values, such as a “Step” slot having a value “Step 2,” andassociated text, “Knead Dough.” Additionally, second text portion 506may include various metadata associated with different features of thesecond step. One such feature may be link data capable of providingsupplemental information to a recipient of the list data for the secondstep. For example, the recipe builder may include links to websiteswithin link menu 510. Link menu 510 may be populated by a builder toinclude information, such as, but not limited to, video, audio, andcontextual information, associated with an action to be performed by thesecond step.

In some embodiments, the links included within link menu 510 andassociated with second text portion 506 may correspond to links to beprovided in response to an expected utterance being spoken. The buildermay input sample utterances within a sample utterance menu 512 that maybe spoken during that particular step. For example, if the step is“knead dough,” some common utterances that may be spoken may include,“How do you knead dough?”, or “Can you show me how to knead dough?” Whensecond step data representing the second step is being rendered and anutterance is spoken, a comparison may be made by an instructionsspeechlet to determine whether to utterance corresponds to one of theexpected utterances. If so, particular link data may be provided to therequesting device. As an illustrative example, when the second step isbeing rendered, an individual may ask “How do you knead dough?” Textdata representing the utterance may be generated and compared with textdata representing the sample utterance. If a similarity score betweenthe two sets of text data is greater than a predefined threshold value,then this may indicate that the utterance spoken is the expectedutterance, and therefore a particular link to a video (e.g.,http:///howtoneeddough/) may be provided to the requesting device.

Further still, second text portion 506 may include item metadataindicating one or more items needed for the instructions. A builder mayinsert the items into item menu 514, as well as a quantity of the itemneeded for the instructions. This way, the instructions speechlet maydetermine items needed for the list, and may be capable of determiningwhether any of those items need to be ordered. In some embodiments,items listed within items menu 514 may be associated with entitiesattributed to slots within NLU system 260 such that NLU system 260 mayresolve text to a particular entity. For example, if an utteranceincludes the text “flour,” then that text may be resolved to the entity“flour” (e.g., the food item).

FIG. 6 is an illustrative flowchart of an exemplary process fordetermining whether an utterance corresponds to an expected utteranceassociated with instruction data representing an instruction of a listof instructions, in accordance with various embodiments. Process 600may, in a non-limiting embodiment, begin at step 602. At step 602,display data representing a GUI including first instruction dataassociated with list data representing a list of instruction may be sentto a first device. For example, display data representing GUI 12including text data indicating a first action that an individual is todo to for a pizza recipe may be sent from computing system 200 toelectronic device 10. Upon receipt, electronic device 10 may beconfigured to display GUI 12 using display screen 212.

At step 604, audio data representing an utterance may be received from afirst device. For example, audio data representing utterance 304 may bereceived by computing system 200 from electronic device 10. In oneembodiment, the utterance may be spoken as the GUI is being rendered bydisplay screen 212. At step 606, text data representing the audio datamay be generated. For instance, upon receipt of the audio data,orchestrator 250 may provide the audio data to ASR system 258, which maygenerate text data representing the audio data.

At step 608, a determination may be made that the list data includesfirst instruction metadata associated with first instruction datarepresenting a first instruction of the list of instructions. Forexample, a recipe speechlet rendering the recipe data may determine thatthe recipe data includes first step data representing a first step(e.g., “Knead dough”). The recipe speechlet may further determine thatthe recipe data includes first step metadata associated with the firststep data. At step 610, the instruction speechlet may determine that thefirst instruction metadata indicates an expected utterance associatedwith the first instruction data. In one embodiment, the firstinstruction metadata may correspond to one or more utterances that couldbe spoken when the first instruction data is being rendered. Forexample, a sample expected utterance may be “How do you knead dough?”Additional first instruction metadata may correspond to links to videosdetailing how to perform an action associated with the first instructiondata such that, if the sample expected utterance is determined to havebeen uttered, that video link may be provided to electronic device 10.At step 612, expected utterance text data representing the expectedutterance may be determined. For instance, the recipe data may includethe expected utterance text data representing the expected utterance.

At step 614, a similarity score may be determined for the text data andthe expected utterance text data. The similarity score may indicate howsimilar to the utterance that was spoken is to an expected utteranceassociated with the first step data. At step 616, a determination may bemade as to whether the similarity score is greater than a predefinedsimilarity score threshold. If, at step 616, it is determined that thesimilarity score is greater than the predefined similarity scorethreshold, then process 600 may proceed to step 618. At step 618, aresponse associated with the expected utterance may be determined. Forexample, the response may indicate that a link to a video associatedwith the first step (e.g., a “how to” video) is being provided. At step620, an action may be caused to occur that is associated with theresponse. For instance, the link to the video may be sent to electronicdevice 10. In some embodiments, if the similarity score is determined tobe greater than the similarity score threshold, then this may indicatethat the utterance is an expected utterance, and therefore a pre-setrule is to be implemented. For example, the pre-set rule may be toprompt a particular video to the requesting device, however persons ofordinary skill in the art will recognize that this is merely exemplary.

If, however, at step 616, the similarity score is determined to be lessthan or equal to the predefined similarity score threshold, then process600 may proceed to step 622. At step 622, the utterance may bedetermined to be unassociated with the expected utterance. At step 624,the text data may be provided to NLU system 260 such that naturallanguage understanding processing may be continue to be performed to thetext data. In some embodiments, the text data may, in parallel todetermining the similarity score, be provided to NLU system 260. In thisparticular scenario, NLU processing may cease if the similarity score isdetermined to be greater than the similarity score threshold, howeverNLU processing may otherwise continue.

The various embodiments of the invention may be implemented by software,but may also be implemented in hardware, or in a combination of hardwareand software. The invention may also be embodied as computer readablecode on a computer readable medium. The computer readable medium may beany data storage device that may thereafter be read by a computersystem.

The above-described embodiments of the invention are presented forpurposes of illustration and are not intended to be limiting. Althoughthe subject matter has been described in language specific to structuralfeature, it is also understood that the subject matter defined in theappended claims is not necessarily limited to the specific featuresdescribed. Rather, the specific features are disclosed as illustrativeforms of implementing the claims.

What is claimed is:
 1. A computer-implemented method comprising:receiving, from a first device associated with a user accountcorresponding to a home, input audio data representing an utterance;performing speech processing using the input audio data to determine theutterance corresponds to an intent; determining the intent correspondsto a plurality of operations including a first operation and a secondoperation; determining the user account is associated with a seconddevice in the home configured to perform first functionality associatedwith the first operation; determining the user account is associatedwith a third device in the home configured to perform secondfunctionality associated with the second operation; causing the seconddevice to perform the first functionality; and causing the third deviceto perform the second functionality.
 2. The computer-implemented methodof claim 1, further comprising: determining data representing the firstfunctionality and the second functionality; using the data to performspeech synthesis to determine output audio data indicating that thefirst functionality and the second functionality have been performed;and sending the output audio data to the first device.
 3. Thecomputer-implemented method of claim 1, further comprising: performingnatural language understanding processing to determine the intentcorresponds to the first operation and the second operation.
 4. Thecomputer-implemented method of claim 3, further comprising: determiningthe utterance corresponds to a first slot and a second slot; determiningthe first slot corresponds to the first functionality; and determiningthe second slot corresponds to the second functionality.
 5. Thecomputer-implemented method of claim 1, wherein determining the intentcorresponds to the plurality of operations comprises: using user profiledata to determine the intent corresponds to the first operation; andusing the user profile data to determine the intent corresponds to thesecond operation.
 6. The computer-implemented method of claim 1,wherein: causing the second device to perform the first functionalitycomprises sending command data to the second device to cause the seconddevice to perform the first functionality.
 7. The computer-implementedmethod of claim 6, further comprising: determining metadata indicatingthat the first operation corresponds to a first device type configuredto perform the first functionality; determining the second devicecorresponds to the first device type; and determining the command databased at least in part on a portion of the metadata corresponding to thefirst functionality.
 8. The computer-implemented method of claim 1,further comprising: determining the plurality of operations furthercomprises a third operation; determining that there is no device capableof performing the third operation; determining data representing thethird operation; using the data to perform speech synthesis to determineoutput audio data indicating that the third operation is to be performedby a user; and sending the output audio data.
 9. Thecomputer-implemented method of claim 1, further comprising: determiningthat a first time period has elapsed without receiving data indicatingthat a third operation has been performed; and causing the second deviceto perform third functionality modifying the first operation.
 10. Thecomputer-implemented method of claim 1, wherein: the second devicecomprises a light; and the first functionality comprises turning on thelight.
 11. A system comprising: at least one processor; and at least onememory comprising instructions that, when executed by the at least oneprocessor, cause the system to: receive, from a first device associatedwith a user account corresponding to a home, input audio datarepresenting an utterance; perform speech processing using the inputaudio data to determine the utterance corresponds to an intent;determine the intent corresponds to a plurality of operations includinga first operation and a second operation; determine the user account isassociated with a second device in the home configured to perform firstfunctionality associated with the first operation; determine the useraccount is associated with a third device in the home configured toperform second functionality associated with the second operation; causethe second device to perform the first functionality; and cause thethird device to perform the second functionality.
 12. The system ofclaim 11, wherein the at least one memory further comprises instructionsthat, when executed by the at least one processor, further cause thesystem to: determine data representing the first functionality and thesecond functionality; use the data to perform speech synthesis todetermine output audio data indicating that the first functionality andthe second functionality have been performed; and send the output audiodata to the first device.
 13. The system of claim 11, wherein the atleast one memory further comprises instructions that, when executed bythe at least one processor, further cause the system to: perform naturallanguage understanding processing to determine the intent corresponds tothe first operation and the second operation.
 14. The system of claim13, wherein the at least one memory further comprises instructions that,when executed by the at least one processor, further cause the systemto: determine the utterance corresponds to a first slot and a secondslot; determine the first slot corresponds to the first functionality;and determine the second slot corresponds to the second functionality.15. The system of claim 11, wherein the instructions that cause thesystem to determine the intent corresponds to the plurality ofoperations comprise instructions that, when executed by the at least oneprocessor, further cause the system to: use user profile data todetermine the intent corresponds to the first operation; and use theuser profile data to determine the intent corresponds to the secondoperation.
 16. The system of claim 11, wherein the at least one memoryfurther comprises instructions that, when executed by the at least oneprocessor, further cause the system to: cause the second device toperform the first functionality comprises sending command data to thesecond device to cause the second device to perform the firstfunctionality.
 17. The system of claim 16, wherein the at least onememory further comprises instructions that, when executed by the atleast one processor, further cause the system to: determine metadataindicating that the first operation corresponds to a first device typeconfigured to perform the first functionality; determine the seconddevice corresponds to the first device type; and determine the commanddata based at least in part on a portion of the metadata correspondingto the first functionality.
 18. The system of claim 11, wherein the atleast one memory further comprises instructions that, when executed bythe at least one processor, further cause the system to: determine theplurality of operations further comprises a third operation; determinethat there is no device capable of performing the third operation;determine data representing the third operation; use the data to performspeech synthesis to determine output audio data indicating that thethird operation is to be performed by a user; and send the output audiodata.
 19. The system of claim 11, wherein the at least one memoryfurther comprises instructions that, when executed by the at least oneprocessor, further cause the system to: determine that a first timeperiod has elapsed without receiving data indicating that a thirdoperation has been performed; and cause the second device to performthird functionality modifying the first operation.
 20. The system ofclaim 11, wherein: the second device comprises a light; and the firstfunctionality comprises turning on the light.